Well, we are in our brand new house. Big thanks to everyone who helped, and to those that ended small group early to help (thanks Barny!) even though we didn’t end up needing the extra people.
It has been a crazy week for me. My boss took off this morning for his annual trip to Taiwan. So Tuesday I started prepping to cover some of the stuff that was upcoming that he would normally handle (everyone does a little). No biggie, except I wan only in for a half-day on tuesday so I could leave and close on the house. Well apparently, at about 5:30 on tuesday our server blue-screened (that is windows admin lingo for took a collosal shit).
They managed to get by without calling me until 7:45 in the morning on Wen. Then began the real fun. Apparently our main internet / network gateway server that is our domain controller and file server as well had some kind of file-system corruption happen to it. This lack of a domain controller started to confuse our app server that drives our main manufacturing software (i.e. no one can enter orders, or buy parts, or do quotes). The confusing part is that server is supposed to be set up to work correctly in the absence of a domain.
Well, the guy in charge of all that stuff (or sorta in charge….there is some confusion here as to who is doing what) got internet back up, and recovered some files. We set up a test-bed for the old drive, booted a linux live-cd, and recoved as much stuff as we could just by using linux’s aggressive mounting capabilities, and then installed windows on another drive on the test-bed and began running some recovery software to get the rest of it.
Meanwhile, I get the app server to start to respond and work correctly with most people’s computers. Except, there are about 4 people who installed windows after we set up the domain controller, and for some reason those 4 people can’t access the application even though they can see the server, and start the app. (it crashes when you try to log in)
So, now I’m building a new server on a second test station based on Windows Server 2003 Small Buisness Server. And trying to learn how to set up a domain all by myself, and do it behind our basic ICS (internet connection sharing) / DHCP server. Then, when I get the DHCP, DNS, and domain portions working correctly, we will take the restored data from the drive recovery going on on the first test station, and try to take the network done for a little while, and bring it back up live with this new server in place, and the new server will be set up to do RAID 1, with extra redundancy.
Down the road we are going to prevent this by building to identical servers, both with 5 disk raid 5 using SAS drives, and use dfs to sync the two arrays so they are always identical. One will be the primary domain controller, and the other will be a secondary. One takes a shit, the other steps up and takes over….no down time, no crisis for the people here, none of whom are technically IT experts.
So, between that, moving and trying to schedule my other normal things, I’ve been more than a little crazy this week.