Towards the end of 2011, my colocated server, Gibson (II) started experiencing what I assumed was hard drive issues. This is the work log for the replacement of the machine.
The server ("Gibson") which normally hosts the website you are looking for is currently offline.
After roughly 5 years of service, Gibson is experiencing significant drive failures and needs down time to be replaced.
I will keep a work log here of progress on the rebuild (all updates are in reverse chronology):
- 2012-05-08
UP AND RUNNING!
If you are still seeing this, give it some time. DNS needs to propogate
RAID system repaired, Gibson2 image booting, functionality to be restored soon. *cross fingers*
- 2012-05-07
So close. Datacenter people reseated some cords and all drives show up now, so RAID has been restored. However, the KVM has ceased working at this point, which is a problem. Trying to get that sorted out.
- 2012-05-06
Ugh. Got the KVM working and one of the drives is not showing up in the RAID controller. My guess, at this point and would be a best case scenario, is that a cable came loose during shipping. Really hope that's the issue.
- 2012-05-05
Waiting to hear back from datacetner about the server. In the mean time, I'm reconditioning what's left of Gibson 2. It had 4 drives in it: 2 x 150GB WD Raptors, and 2 x 400GB WD RE2's (Raid Edition). All but one Raptor are dead. (They make the most wonderfully horrible clanking noises if you try to read from them.) The fact that Gibson even booted and worked at all for the last however-long these drives were out is a testament to the RAID card it had. Sheesh.
- 2012-05-04
The server has been racked and turned on -- and the RAID is missing. And the KVM (remote management) is not responding. Not sure what's going on yet.
- 2012-05-03
Server has arribed at datacenter a day early. Waiting to hear back on when it will be re-racked.
- 2012-05-02
UPS tracker says server is in TX at airport.
- 2012-05-01
Server in UPS hands. Off it goes, should arrive at DC on Friday.
- 2012-04-30
Work day; got SSL certs for remote management purchased, installed, and tested. Setup ip addresses for DC network. Zip-tied up all the interior cords nice and snazzy, secured the top cover, put it in the shipping bag, moved the cats, and boxed it up. It'll hit the road wednesday morning. Probably get to the DC by saturday. Hopefully come sunday we'll all be looking at normal websites again. At which point I can start the long process of upgrading the OS from Fedora 6 to something not made 3 years ago.
- 2012-04-29
Added commenting ability below.
Very excited about this screenshot:
Click to enlarge
What you are looking at is my ability to completely, remotely control the server with KVM over IP, plus monitor fan speeds, temperatures gauges, etc. Additionally, the SSD drives and 6TB RAID5 array is running in VMWare. Finally, the single virtual image installed? All of the sites that would normally be hosted here -- ready to go once the machine makes it back to the DC (and after a little network configuration repair). This is very exciting.
100% formatted, total: 57h 01m
- 2012-04-28
92% complete @ 51h 04m -- Finally going to wrap up sometime while I sleep.
Discovered that old Gibson, while supporting PWM (variable speed) fans, was built with 3 pin fans instead -- so they've been cranking full speed for the last 5 years. Ooops. I ordered new 4 pin fans to replace the old ones to reduce noise as I am recommissioning it as my home server.
86% complete @ 46h 18m
71% complete @ 37h 40m
- 2012-04-27
44.7% after 22 hours. Looks like my jibe about it "taking 40 hours" was closer to accurate than I thought.
- 2012-04-26
Initializing RAID5 array. This should only take, like, 40 hours update: estimate sits at 24 hours
RAID card successfully integrated with VMWare ESXi5
Moved the raid card from G2 to G3 and getting it rebuilt. Considering getting the raid card bios up to date.
The drive copy went smashingly. The old gibson OS has no idea its now living in a virtual wonderland -- outside of resetting its ethernet adapter. Database checked out fine, files all present. Yay!
- 2012-04-25
This things fans are intense. Hard drive image copy underway -- once the image is stored, it will be moved to the new server and started up as a virtual machine.
Server just arrivd back. Yay! I expected fully evolved, walking, talking dust bunnies but was surprised to find the chassis almost dust free. Naturally, the cats approved of this shipment.
- 2012-04-24
Gibson has landed in San Jose. Should be on a truck to my place tomorrow.
- 2012-04-23
The server is on a plane from Texas to California. Flyyin, on a jet plane. Don't know when I'll boot up agaaaaiiin.
- 2012-04-22
Got the server scheduled for pickup on Monday, delivered from Texas to CA by Wednesday.
- 2012-04-21
Gibson has been unracked, packed up, and will be getting shipped back to me tomorrow. Will take a few days to get here.
- 2012-04-20
The server migration/upgrade has begun. If you are seeing this then Gibson is no longer serving requests for the website you're looking for.
- 2012-03-09
Gibson has been holding up quite well over the last few days, actually, none-the-less, the replacement build is nearly complete. At this point, I am only waiting on the cpu heatsink/fan parts to come in. All drives have been wired, fans hooked up, etc.
- 2012-03-05
Gibson looks like it ate dirt today.
Yeah, crashed, but came back up. Hopefully it'll hold for another few weeks while the new server is pieced together.
- 2012-03-01
The majority of the parts are in. Still missing heat sinks, some moliex->esata power adapters. Everything I have so far is installed$
- 2012-02-25
New parts ordered. Dual Xeon motherboard, two 5606 processors, 64GB ram, 4x2TB WD HD, 2x32GB SSD, Chenbro 3u rackmount.
]]>
Ah, what a good movie. The artist responsible.]]>
]]>
]]>
