I picked up the server this morning and got it up and running at its new home (i.e. my home). It’s not a pretty situation: it sits on top of a wire shelf (strong enough to support it, thank you), the cabling around it is a cable maintenance guy’s nightmare, and there is now a constant hum in my basement (the server literally sounds like a jet engine when it runs; I’ll take a short video clip of it when I reboot it next). But it works…

Amazingly, the server booted right up and I logged right in without a problem. I looked at the event log and I found out why the server shut down after the guys rebooted it. Apparently, another server running Small Business Server (SBS) was put on the same network as mine. This is a problem because if SBS sees another SBS install on the same network, it sees it as a licensing violation, and the server shuts down in an hour. This means that at lease some of the unexpected shutdowns and the final shutdown weren’t even the fault of my server. That, however, is an issue I will take up with the ISP when the opportunity arises.

I am running a full server diagnostic right now (thanks, Dell OpenManage). I am also moving big files to the external USB drive to keep things clean. Unfortunately, the server apparently has USB 1.1 and not 2.0, which means very slow speeds, but it’ll be worth it in the long run.

I’ve already got network connectivity to the server, and things seem to be fine. I shut down all sims and database services as soon as I started the server, so we won’t get any unexpected events. I’ll address game situations once I know the server is healthy.

More details later tonight — but it looks quite probable that we’ll be up by Monday.

Since I am moving the server out of the colocation facility and back into my home (thanks to a business-grade FIOS line), there are some side benefits I can take advantage of — all of which revolve around the fact that the amount of infrastructure added to the game is only limited by the amount of power I can feed from my circuit breaker, and how many servers I can fit in my basement without generating enough heat to turn off my gas service. (Then again, considering it’s about 7 degrees in New York right now, that may not be a bad idea!)

In short, the following are all possibilities now that we have greater flexibility.

• Offloading data to external USB drives. Obviously this would not be live game data; it would be the backups of that data. This will help the server disk subsystem a lot by reducing fragmentation and load during backup times. This is a no-brainer; I have a 300GB USB drive sitting under my desk waiting.
• Firing up the old CSFBL server to handle the web services. Those that remember CSFBL 2 1/2 years ago would have remembered our old server. It’s actually not a bad server (dual processor, 2GB RAM); it’s just big (coffee-table tower). It’s definitely less powerful than the current server, so we will do the logical thing: put the web site on the less powerful (old) server, and put the database and sim engine on the more powerful (current) server. This could have a significant performance improvement, especially considering the load that the forums puts on the current server (to the tune of 500+ MB of RAM). The only delay in doing this will be if I need to get more electrical juice in my basement storage room to handle two servers demanding some 400 watts of power each.
• Getting another server from babarfloyd. Rick (babarfloyd) has often offered to provide a server he has in his house to help the game, and with the recent turn of events, has even offered to ship it to me. This could, in theory, mean three servers, but we’re getting ahead of ourselves. I’d definitely need to consider both power and cooling demands if I throw three servers in my basement (not counting an electric bill that would start to grow quite quickly), and there are programming and logistic issues to address before we can scale out this far, but it’s good to have this as an option.

Of course, all this could be thrown out the window if the FIOS Internet connection proves unreliable or lacking the throughput we truly need. If this is the case, the clear choice is to eventually move to managed server hosting. I’ve spoken to people at SoftLayer, who have an excellent reputation (I did a lot of research) and who were also very quick to answer all my inquiries. They provide managed dedicated servers — basically, they make sure it runs 24/7, you worry about what runs on it. The downside is the cost — $300+ per month per server — but this is the highest-quality hosting we can get, and is reasonable considering the massive data and processing needs of CSFBL.

We can’t predict where we’ll be in a year, but the possibilities outlined above are all things that may become reality over the coming weeks and/or months.

I just got off the phone with the ISP. They are having someone pack up the server and ship it to me. As soon as I have a tracking number, I’ll post it here so everyone can track it’s progress.

I wanted to give everyone an update on what we’re doing this week to get things back on track.

• Last Friday, Verizon came and upgraded my FIOS Internet connection to a business-grade line. Of course, by the time I got around to checking it (11:30PM on Friday night), it didn’t work. Fortunately, 30 minutes on the phone with Verizon fixed the issue. If all other things are as they seem and as Verizon tells me they are, this line is perfectly adequate for hosting web pages like CSFBL.
• A little more information on the server itself… Apparently it won’t boot at all (it gives an operating system error during boot). Hopefully, only some core operating system file is corrupt, and there is no larger data loss. Still, recovery could take hours once I get possession of it.
• I’ve got a few options on getting physical possession of the server, and none of them unfortunately are foolproof, and all of them require 3-4 hours of my time during the day which is a logistical impossibility considering I work during the week. I’m going to have a conversation with the manager of the datacenter that the server is located in today to discuss options for getting it that work within the constraints of our mutual schedules.

I’ll post an update after I find out more information today.

Just a couple of things:

• Well, if Verizon did what it was supposed to do, Brian’s FiOS line should have been upgraded to business class sometime between 8am-12pm. The first piece of fixing this should be complete then.
• Waiting to hear from Brian on if he has a tentative date for picking up the server from our ISP.
• I would like to thank all of you who maintained a good atmosphere on this blog. Please note that just because some people’s opinion doesn’t match your own, they are allowed to express it. We are all for free speech.
• However, a few days ago, I was forced to moderate three users on this blog due to their vulgar and combative posts. Their intent from post #1 was to cause trouble (one of them even thought it was funny to use the name of the dictator who ran Germany in WWII). They were given a chance and lost it. One of them, noticing that they had been moderated, tried to say he will report me. Go figure.

We are at two weeks now since the CSFBL went down. We have two major hurdles left to go: picking up the server and figuring out what went wrong with it. We will get there soon, I promise.

- Chris G.

On Friday, February 2, between 8AM and 12PM, Verizon will be shutting down my residential FIOS Internet connection and convert me to a business-grade FIOS Internet connection. This will provide me with the ability to host web sites out of my home.

The next step is to schedule a time when I can get physical access to the server, remove it from the datacenter it is currently in, and bring it home. Before I do that I have to clear out some space in my basement for it; I have a storage room that I should be able to find room in. I also have to make sure I have sufficient electrical capacity to run the server and related devices; if this proves to be a problem, I have to get an electrician to throw a new breaker in the circuit box. (I do some electrical work, but I don’t do things to that level!)

I’m in the process of figuring out when I can get access to the datacenter without paying another $125 (or $150, depending on the price du jour). The ISP told me that if someone is already on-site they would not charge, so I’ll try to find a day where someone is on site and schedule the extraction for that day. One of the members of our community, rewster, has offered to give me a hand in the operation (something that’s needed, as moving a rack server in a busy Manhattan street is no simple task). I’d much rather give him gas and toll money and buy him lunch than give the ISP $125 (or $150) to sit with their thumb up their rump…

When I find out more I’ll let you know. Keep your fingers crossed that things will come together soon!

I wanted to say before I went to sleep the following.

Thank you for all of support. This hasn’t been a thrilling past couple of days for any of us, especially Brian. It really bothers him what has happened with the server. Whenever you have something special you created, you hate to see it break in front of your eyes.

I just sent an e-mail to Brian, the developers, and the community managers that basically summed up the 100+ ideas in the Post-Reboot post. I am curious to see what all of us on the ComputerSims staff think about the wonderful ideas you all presented. There are lot of ideas we haven’t thought about before that many of you made interesting cases for.

I don’t know what is in store for the next few days, weeks, months, etc. We will keep you up-to-date, no matter what. Decisions are probably going to be made fairly quickly, so we should be able to give you a general idea on what direction we are going soon. We have to move fairly quickly too to keep our user base together. The longer we wait, the harder it is to get things back to normal.

I also asked Brian if he wants a donation info on here. I am sure he will talk about that tomorrow.

All I know is, I miss the game already. We all do…

- Chris G.

After a 10 minute discussion with a guy who I think is the manager of the network operations center (NOC), I finally got a guy to go down a reboot the server for the price of $125.00. The discussion that took place does not deserve further mention. The guys I work with overheard my half of the conversation and couldn’t believe what I was saying, so you can imagine what was coming from the other end of the receiver.

Anyway, the server rebooted, and I was able to get a login prompt again. (All remote work is done using Microsoft’s Remote Desktop.) I entered a username and password, then found the “Applying computer settings”window (or whatever Microsoft calls it). I waited a minute, and then the window abruptly closed.

I tried logging back in, and got the same results. Hmm…

I tried logging in a third time using a different user account, and this time got a message: “The system is shutting down.” About ten minutes later, the server was totally unresponsive — it doesn’t even respond to a ping request.

What does this mean? I have to get physical access to the server console to do anything at this point. This is not so easy these days, as I have a day job and the ISP is surely not going to have a guy sit around with me at 9PM at night while I fix a server — at least not without charging $125/hr.

The other solution is to simply pick up the server from the ISP and bring it home to work on it. For reasons explained above, this is the more viable solution. Once I get it, I can assess the damage and figure out what to do.

I’m going to contact the ISP (I’ll deal directly with the NOC manager at this point) and find out if there’s a time I can remove the server without having to pay $125/hr for someone to open a door for me. Hopefully I can arrange something with them that works with my schedule. If not, I’ll see if I can find someone to pick it up for me (I may have a few people who owe me a favor or two).

What do I think the damage is? I think the data is intact; certainly the database likely is. I’d bet that Windows is hosed. The disks were in a fault-tolerant RAID5 configuration, so it’s unlikely that we’d experience any kind of major data loss.

The options once the server comes back up:

1. Fix the server and put it back in the NOC. The pros with this: we’re back where we started. The cons: we’re back where we started. This setup costs about $275/mo. (not counting $125/hr fees for reboots), about 2/3 of which is the server lease, which is up in August 2007.
2. Fix the server and move the game to a dedicated server at a different ISP. I spoke with people at SoftLayer a while back, and they would certainly be a top consideration. The pros: excellent reputation and they manage the hardware. The cons: about $380/mo. for the server. Granted the server is more powerful than what we have today, but that’s close to $5,000 a year for a web site that brings in about $2,500 a year. You see the dilemma.

We’re at a crossroads, folks, and I need your feedback. Keeping CSFBL running is something I can no longer fund out of my pocket. I want to keep the game running, but I can’t go in without having a reasonable level of confidence that I’m not going to be out $2,500 per year.

Considering all this, how interested or willing would you be with the following proposals?

1. Player renames can be purchased for as low as $1.00. (Players with one or more years experience would cost more, on a sliding scale.)
2. Trades would only be allowed between players who have donated at least $5.00 in the past 6 months, or $10.00 in the past 12 months. (Credit would go back to the start of 2006.)
3. >> What are your suggestions? Let us know and I’ll add them here. <<

I’d like to hear other suggestions for features that you guys would be willing to pay for. I’m looking towards adding features that add fun and functionality to the game — and that you’d be willing to pay a nominal fee for. I don’t want you to give something for nothing, but the things asked for should be reasonable.

Thanks for your continued support and patience.

Are you curious as to what the error message that comes up when you try to log in to the server? Here it is, in all its evil Microsoft glory…

Such a typical Microsoft error. I remember the first time I saw the legendary “memory could not be ‘read’” message, and just loved it. Is that the best description of an error that Microsoft could come up with?

I did a quick Google search for this error. Although nothing was conclusive, some posts do not sound very reassuring.

Still no word back from my ISP. I’m going to tell them to just go ahead and do it and bill me for the hour. Let’s see what happens. At this point I need to know whether or not I should be scheduling time to take delivery of the server in order to repair it and figure out what the next step is going to be.

Not much to report beyond what Chris reported earlier. My ISP hasn’t gotten back to me, and I don’t expect them to do so until tomorrow. So much for service.

These issues bring a number of larger CSFBL woes to light. We’ve talked a lot about these in the past, and I do want to take this opportunity to talk about them more… Just not right now. Let’s see what happens tomorrow morning, and we’ll take it from there.

Thanks to everyone for their patience. I’ll keep you in the loop as best I can!