Server restore in progress

I picked up the server this morning and got it up and running at its new home (i.e. my home). It’s not a pretty situation: it sits on top of a wire shelf (strong enough to support it, thank you), the cabling around it is a cable maintenance guy’s nightmare, and there is now a constant hum in my basement (the server literally sounds like a jet engine when it runs; I’ll take a short video clip of it when I reboot it next). But it works…

Amazingly, the server booted right up and I logged right in without a problem. I looked at the event log and I found out why the server shut down after the guys rebooted it. Apparently, another server running Small Business Server (SBS) was put on the same network as mine. This is a problem because if SBS sees another SBS install on the same network, it sees it as a licensing violation, and the server shuts down in an hour. This means that at lease some of the unexpected shutdowns and the final shutdown weren’t even the fault of my server. That, however, is an issue I will take up with the ISP when the opportunity arises.

I am running a full server diagnostic right now (thanks, Dell OpenManage). I am also moving big files to the external USB drive to keep things clean. Unfortunately, the server apparently has USB 1.1 and not 2.0, which means very slow speeds, but it’ll be worth it in the long run.

I’ve already got network connectivity to the server, and things seem to be fine. I shut down all sims and database services as soon as I started the server, so we won’t get any unexpected events. I’ll address game situations once I know the server is healthy.

More details later tonight — but it looks quite probable that we’ll be up by Monday.

The Eagle has Landed

I received a phone call this morning notifying me that the server has been delivered and is sitting in a rather large box in the office. I will pick it up over the weekend and get to work on it — expect a post here when I have more details.

A few notes on the delivery

Just in case you didn’t see my comment on the earlier post:

Brian is having the server delivered to his father’s office in NY. So the odds are that someone will be there when UPS arrives (unless UPS shows up at an odd hour). Given that the delivery is within the NYC area, it would be surprising if it took more then a day or two to arrive.

Once he has the server, he has to unpack it, hook it up, find something called free time, and figure out what the heck went wrong. So don’t expect an immediate answer a few hours after it arrives…but hopefully we will have some news coming your way by the end of the weekend.

- Chris G.

Outlining the “new” infrastructure

Since I am moving the server out of the colocation facility and back into my home (thanks to a business-grade FIOS line), there are some side benefits I can take advantage of — all of which revolve around the fact that the amount of infrastructure added to the game is only limited by the amount of power I can feed from my circuit breaker, and how many servers I can fit in my basement without generating enough heat to turn off my gas service. (Then again, considering it’s about 7 degrees in New York right now, that may not be a bad idea!)

In short, the following are all possibilities now that we have greater flexibility.

• Offloading data to external USB drives. Obviously this would not be live game data; it would be the backups of that data. This will help the server disk subsystem a lot by reducing fragmentation and load during backup times. This is a no-brainer; I have a 300GB USB drive sitting under my desk waiting.
• Firing up the old CSFBL server to handle the web services. Those that remember CSFBL 2 1/2 years ago would have remembered our old server. It’s actually not a bad server (dual processor, 2GB RAM); it’s just big (coffee-table tower). It’s definitely less powerful than the current server, so we will do the logical thing: put the web site on the less powerful (old) server, and put the database and sim engine on the more powerful (current) server. This could have a significant performance improvement, especially considering the load that the forums puts on the current server (to the tune of 500+ MB of RAM). The only delay in doing this will be if I need to get more electrical juice in my basement storage room to handle two servers demanding some 400 watts of power each.
• Getting another server from babarfloyd. Rick (babarfloyd) has often offered to provide a server he has in his house to help the game, and with the recent turn of events, has even offered to ship it to me. This could, in theory, mean three servers, but we’re getting ahead of ourselves. I’d definitely need to consider both power and cooling demands if I throw three servers in my basement (not counting an electric bill that would start to grow quite quickly), and there are programming and logistic issues to address before we can scale out this far, but it’s good to have this as an option.

Of course, all this could be thrown out the window if the FIOS Internet connection proves unreliable or lacking the throughput we truly need. If this is the case, the clear choice is to eventually move to managed server hosting. I’ve spoken to people at SoftLayer, who have an excellent reputation (I did a lot of research) and who were also very quick to answer all my inquiries. They provide managed dedicated servers — basically, they make sure it runs 24/7, you worry about what runs on it. The downside is the cost — $300+ per month per server — but this is the highest-quality hosting we can get, and is reasonable considering the massive data and processing needs of CSFBL.

We can’t predict where we’ll be in a year, but the possibilities outlined above are all things that may become reality over the coming weeks and/or months.

The package is being delivered

I just got off the phone with the ISP. They are having someone pack up the server and ship it to me. As soon as I have a tracking number, I’ll post it here so everyone can track it’s progress.

What’s happening this week

I wanted to give everyone an update on what we’re doing this week to get things back on track.

• Last Friday, Verizon came and upgraded my FIOS Internet connection to a business-grade line. Of course, by the time I got around to checking it (11:30PM on Friday night), it didn’t work. Fortunately, 30 minutes on the phone with Verizon fixed the issue. If all other things are as they seem and as Verizon tells me they are, this line is perfectly adequate for hosting web pages like CSFBL.
• A little more information on the server itself… Apparently it won’t boot at all (it gives an operating system error during boot). Hopefully, only some core operating system file is corrupt, and there is no larger data loss. Still, recovery could take hours once I get possession of it.
• I’ve got a few options on getting physical possession of the server, and none of them unfortunately are foolproof, and all of them require 3-4 hours of my time during the day which is a logistical impossibility considering I work during the week. I’m going to have a conversation with the manager of the datacenter that the server is located in today to discuss options for getting it that work within the constraints of our mutual schedules.

I’ll post an update after I find out more information today.

ComputerSims Baseball – Some answers to your questions

The following is written by myself and Babarfloyd about ComputerSims Baseball, the successor to the CSFBL. Some longtime CSFBL users might already know this information, but I know we have many new users around as well. Also Babarfloyd posted his parts as comments to other blog posts, so they may have been missed.

Here it goes…

Question – What is ComputerSims Baseball?
Chris G. - To put it simple, ComputerSims Baseball is what the CSFBL should have been from the start. It is a complete rewrite of the game that is designed to be much faster simming, more features, more flexibility, better stability, and better scalability. Many of the issues that you see in today’s CSFBL are going to be eliminated by the new game.

Question: Why a rewrite? I love the current game!
Chris G. – The current CSFBL is designed for just one tenth of the traffic it has today (mostly due to Brian having no idea the CSFBL would get this popular). It simply has reached the breaking point (and some might argue, exceeded it). All the recent sim problems with the CSFBL are the result of this old and poorly written code. Not to mention, there are many quirks in the game (stealing logic and base running logic for example) that drive everyone crazy and are long overdue to be rewritten. Finally, there are a lot of features we would like to add, but simply can’t (or can’t easily) in the current game.

Question: That could be an opportunity for donations as well. There was talk that not every league would be migrated over to the new engine, so maybe only leagues that contribute a certain amount can go?
Babarfloyd: I doubt we will be porting over leagues that have low ownership but I’m pretty sure most leagues will be ported over. I’m pretty sure Brian already wrote some code that will port leagues over.

Question: What sort of features will be in ComputerSims Baseball?

• It will sim one play at a time per game then go on to the next game and sim one play and so on. This will allow you to “watch” any game in real time and it will also allow you to “manage” in real time. You will be able to show up in the 3rd inning and manage until you have to leave or until the end of the game. When you are not there to manage or you are there but you aren’t making any decisions in the allowed time (maybe 30 seconds between plays) the sim will use your default manager options.
• Much much much faster and more reliable
• The stealing logic and base running logic is GREATLY improved.
• There will be more manager options, too. Things like I mentioned above (hit and run, run and hit, double/triple steals, etc) plus things like defense positions (double play depth, guard the lines, infield in, deep outfield, shifts, etc).
• We are planning on allowing tournaments and head to head exhibition games at any time between 2 owners or you against the CPU. There will be options to use any team from your franchise (pick the year) or create your own team from any players from any year in your entire Franchise history…basically a Franchise All-Star Team.
• Another feature would be to schedule league games. Say the next sim is scheduled at noon and my team is going to play Realm’s. We will have up to 11:59 to reschedule the game between noon and 2:00 so that we could mange in real time against each other.
• There will be a live draft with a queue like we did for Moneyball. You will have the option of creating a new league and having a live draft instead of just populating 24 teams with players.
• Pinch hitting, pinch runner, and double switches (leagues/divisions will have the option of being DH or no DH) will be part of the CSBB.
• Since we are going to allow pitchers to hit, we need to generate batting stats for pitchers now. In fact, we are going to generate batting, pitching, and defensive stats for every player. This will allow teams to throw in a backup LF to pitch in blowout games. Of course this will up the chance of him getting hurt if his pitching stats are poor, which most of them will be, but it’s going to be an option.
• We are also going to have 3 levels of minors: AAA, AA, A. It’s been suggested before and I’m not sure if we’ve agreed on this but it might be possible to switch regular players over to pitchers (it’s done a lot in real life quote a bit).
• The draft architecture will allow us to easily add a Rule 5 draft and a Contraction/Expansion draft. We don’t plan on adding either of these features for the first release, though, and they will be a per league option.
• Contraction/Expansion will be easier to implement in the new game because the new design of the draft can be used for this. I doubt it’ll be available in the first release, though.
• We planned on allowing leagues with 18 teams all the way up to 32 teams…I think, I’m not positive of the numbers but I know there will be different sizes available.
• There will be the ability to trade draft picks and to rate/comment other players on their trading (ebay style ratings). The hope is this will help establish a paper trail that people can follow if they are worried about the person they are trading with. It will also making it much harder for cheaters to get away with lopsided trades.

Question: How fast is the new sim engine that will make up ComputerSims Baseball?
Chris G.: Babarfloyd previously told me that he has simmed a single game in 3 one-hundredths of a second with the new sim engine, while the current CSFBL sims a single game every 5 seconds. Keep in mind that this sim engine is pretty simple at the moment, but even when it is fully written it should easily sim a single game in less then a second…a huge improvement over the current game.
Babarfloyd:CSFBL sims thousands of games every 3 hours and it sims a full game in about 5 seconds and then goes on to the next game. CSBB will have the ability to sim thousands of games at the same exact time and have them all done in under 60 minutes.

Question: When will ComputerSims Baseball be ready?!?!?!
Babarfloyd: Not sure, but we have been making great progress on SimpleSim, the precursor to CSBB. SimpleSim is a simple baseball engine that we’re using to see what works and what doesn’t. For example, a few weeks ago I ported over the Steal Logic from CSFBL and realized there was no way it was going to work with CSBB. I spent a week tinkering with it and then got everyone up to speed on it during our weekly dev chat. Harborpirate took over at that point and reworked it all to fit in with the new architecture. In CSFBL a steal is a full play. The guy steals and either makes or he’s thrown out…that’s it…no hit and run, not run and hit, no double steals, triple steals, runners going on 2 outs, etc. Harborpirate fixed all that and now all of that is possible in CSBB.

Harborpirate also wrote a killer algorithm for base running logic and we’ve been playing around with it for the last week. It is a huge upgrade from CSFBL. Most runners will score from second on a single to the outfield and faster runners will have a better chance of scoring from first.

I’ve been working on creating xml files that contain full rosters for teams. Right now SimpleSim only loads 10 players per team (9 for the lineup with a DH and 1 pitcher). Then I need to code in loading the full rosters to the bench and bullpen.

Chris G: I hope that sometime in 2007, we will have something real to show you all. No promises of course, but that is my hope (and I think all of our hopes)!

Brian: When it is ready!

Couple of things

Just a couple of things:

• Well, if Verizon did what it was supposed to do, Brian’s FiOS line should have been upgraded to business class sometime between 8am-12pm. The first piece of fixing this should be complete then.
• Waiting to hear from Brian on if he has a tentative date for picking up the server from our ISP.
• I would like to thank all of you who maintained a good atmosphere on this blog. Please note that just because some people’s opinion doesn’t match your own, they are allowed to express it. We are all for free speech.
• However, a few days ago, I was forced to moderate three users on this blog due to their vulgar and combative posts. Their intent from post #1 was to cause trouble (one of them even thought it was funny to use the name of the dictator who ran Germany in WWII). They were given a chance and lost it. One of them, noticing that they had been moderated, tried to say he will report me. Go figure.

We are at two weeks now since the CSFBL went down. We have two major hurdles left to go: picking up the server and figuring out what went wrong with it. We will get there soon, I promise.

- Chris G.

A note regarding comments on this blog

As you all well know, anyone can post a comment on this blog. All that is required is typing in an email address and name. Neither of those are verified, so it is realtively easy to be an imposter of someone else.

Note well that the blog stores the IP address of each comment, so although you may think you are anonymous, in practice you are not.

I expect people to behave here with the same level of respect that they have on the CSFBL forums. Clear and blatant abuse will be reported to your Internet Service Provider as appropriate.

(I can’t believe I had to write this post.)

Restore (Phase 1) happening on Friday

On Friday, February 2, between 8AM and 12PM, Verizon will be shutting down my residential FIOS Internet connection and convert me to a business-grade FIOS Internet connection. This will provide me with the ability to host web sites out of my home.

The next step is to schedule a time when I can get physical access to the server, remove it from the datacenter it is currently in, and bring it home. Before I do that I have to clear out some space in my basement for it; I have a storage room that I should be able to find room in. I also have to make sure I have sufficient electrical capacity to run the server and related devices; if this proves to be a problem, I have to get an electrician to throw a new breaker in the circuit box. (I do some electrical work, but I don’t do things to that level!)

I’m in the process of figuring out when I can get access to the datacenter without paying another $125 (or $150, depending on the price du jour). The ISP told me that if someone is already on-site they would not charge, so I’ll try to find a day where someone is on site and schedule the extraction for that day. One of the members of our community, rewster, has offered to give me a hand in the operation (something that’s needed, as moving a rack server in a busy Manhattan street is no simple task). I’d much rather give him gas and toll money and buy him lunch than give the ISP $125 (or $150) to sit with their thumb up their rump…

When I find out more I’ll let you know. Keep your fingers crossed that things will come together soon!