The final story in my little series, really revolves around the dread of all server room administrators... More nerve shredding than fire! More nerve tingling than granting terminal access!.... We are of course talking about moving server room....
The plan was eeked out of management in literally, fifteen minutes of a meeting... We had three 6' racks all the wiring, all 2U servers in one rack with a 32 port switch at the top, nicely wired by good self (wishes I had a picture, but we were still using film for photo's back then, and it was too much effort).
The middle rack was a bit more the experimental kit, there was a 4U server in there, a couple of desktop cased machines stood directly on top of that unit - with plastic cutouts to stop hot air from the back feeding back to the front of the rack - yes, even with ghetto server layouts forced upon me, I did try to make the best of things and keep good order...
The third rack was telecomms units, it contained a bank of modems (yes, 64K modems!) another ethernet switch, an unmanaged switch too, and our SAN.
The SAN was a Dell EquiLogic; from memory I remember having 16 drives in it of 120Gb each, and two were hot spares...
The plan, from management, "unplug it all, load it into the two Nissan Primera Estate company cars, and move it from Derbyshire to Leicestershire"...
That was it, "unplug and move"!
We had two days, over a weekend, to do this... It was Thursday afternoon... The clock was on.
Before you ask, I can't tell you much about why we had to move, nor whom I was working for, but let us just say, a week after the move the company changed name, and I changed employment soon after.
So, Thursday afternoon, I sent out e-mail to everyone, all staff, literally everyone, and said "The Servers are ALL going off, the Network is coming down, ALL IT services will be suspended at 3pm the following day".
Of course the whole place started to moan, so I promptly sent another mail.. "No seriously, ALL Services will be offline at 3pm, I'm serious!" And I left the building to go see this new site... Two hours driving later, it's nearing 5pm Thursday and I arrive at a tin-shack looking warehouse like building, am let in by a man with a carrier bag and a cough. I get the keys off of him to, and go check out the "IT Room" as he calls it.
It's a desk, with a single 15amp 240v rail supply and a bunch of cut off ethernet cables spilling out the wall in one corner.
There is no aircon, there is no patch panel, there is no racking... And it's on the first floor... (If you're from the US and reading this, here in the UK the floor against the ground is called "the ground floor", and the "first floor" is above that, it is up at least one flight of stairs... Yep, stairs... No lift (elevator).
Company credit card out, I head back to the office, and I order on next day (before 11am) delivery two RJ45 crimpers, two spools of ethernet cable, a pair of wall mountable 8U cabinets and a laptop (you know, because I thought I might need one) and a pair of walkie-talkies.
I then survey the kit I have to move, now, I can ignore the modems in the large, they unplug and lift out, but there's no telecomms due at the other end for a fortnight, so they'll go dark, they can be the last thing we move. The switches in that right hand rack though therefore become free, and I can take them our now and start to set up the ethernet at the other end.
I have been assigned a minion from the shop floor to help, Friday morning comes and I await until 1pm that afternoon for my "before 11am delivery"... things are going so swimmingly already. But I have had time to take the two desktop form factor servers offline, the 4U unit off line, the front blanking all out and disassemble the middle rack. I've also got the switches from the right hand telecoms cabinet...
In our first transfer load we therefore have the rack, cables, UPS's, a 1U fold out terminal and crimpers in one car. The two desktop servers and the 4U unit in the other. We drive to the new site, unload the servers onto a trolley and lift them up stairs, placing them in a corner on the floor.
We then pull up the rack, assemble it, and fit a switch to the top of the cabinet and I send my helper off to survey the site, looking for any and all RJ45 sockets, and noting where they are, room by room and any notes on them. I meanwhile start to splice new RJ45 connectors onto every one of the fifty or so ethernet cables laying on the floor, and I plug them into the switch. I braid them all and cap them and wrap them nicely into the rack cable managements either side.
The switch has a useful line-test function, and my minion has the laptop!... So he can just plug a cable in at the other end and I see a light come on... Over the radio I ask him to pick a socket, tell me if it's marked and to plug in... We do this room by room, and I move patches around to group rooms together, and I tell him a number to mark on the socket with a sharpie.
We've been at this mind numbing task for a couple of hours when I decide we need a break, and I go down to meet him... I'm a little aghast to see he's not written on the little window areas for the socket numbers, he's written them in letters as tall as the socket itself... *slaps forehead*.... It looks a mess, but it works, I'll have to come around in a weeks time with some acetone to clean it up, but it'll do... It'll do.
Fifty odd sockets later, we've got the whole building ethernet hooked into this switch. The rack is powered and the UPS's are stable, reporting good, and the 4U server is up and fine from the laptop over a few of the sockets I've checked.
The two desktop servers survive the move too, and they're placed in the footing of the rack, rather then perched ontop.
Everything looks fab, and we've got this all done before 7pm on the Friday! Things are looking GOOOODD!
I buy the chap helping me a pub meal, and he takes the company car home, I expect to see him Saturday morning.
Saturday morning, before switching down everything in the office, I check everyone is offline, and start to kill the servers down. The SAN down, and pulling everything out of the rack, I'm stacking all the patch cables I'm taking all the units out and the car is pretty full. Where is the other car?... 9am... 10am... No sign, I've got to call him.
He's at home, in bed... His wife describes him as, "Drunk as a lord"... He apparently didn't realise I needed him the next day, and had a skin full after he'd gone home. He can't drive.
So, I'm flying solo...
I drive the first load down to the new site, and basically throw it all out the back into the loading dock, and set off for the next load, I get the UPS's for this rack in the car and two of the 2U servers... and it's pretty much at load... Four more trips and I have all the servers at the new site, in the loading bay, and I'm taking the left hand rack down at the old place.
It won't come apart, and it's really a two man job... So, I give up, and I load all the bits I can up, and go to the new site, and decide to just stack the machines up off the floor but in a pile, rather then rack mounted, for a while.
So, machine by machine, I lug them up the stairs... It's heavy, it's hard, and I'm on the penultimate machine... When half way up the first flight of stairs I feel I'm about to loose my grip.
I try to catch a better hold, but it all happens at once, the machine is free, I'm toppling to my side as the weight takes me, and I'm going down the stairs on my side with my left finger in the thumb hook on the front of the server.
Lets just say, it was my left finger... And the server fell to my right.... Yeah, snappy snappy.
So, I have definitely broken my finger. And I've definitely broken the server, as it bounced down about eight hard steps, the real flanges on the rack rails are all bent, the handles on the PSU units are smashed and the drives have shook loose...
Swearing and hobbling I crawl over to the machine, pop the lid and it's all fucked, just properly fucked. The motherboard has popped up, everything is a mess.... That's an expensive mistake, but not-one I've ever admitted to, until now.
My finger however, needs attention, so I lock up and take a trip to A&E... After three hours they basically say it's a bad break, and put my hand in a mini-cast across the back of my hand and down my finger itself its all fixed. I can use my wrist.
I return to the site, and luckily for me, there are no more servers to be lifted upstairs... Because someone has hiked open the loading dock roller door and pinched the one remaining server which was sat there.... They've also taken the laptop from upstairs, the two desktop profile servers and a bunch of other stuff...
When I go talk to the "security" guy, the old geeza with the bag from the previous day, he's still carrying his bag... I ask if he saw anything, "yeah saw one of your lads moving your computer things"... I explain he wasn't one of our lads... "I thought that funny, he didn't lift the roller, but crawled under it"... Total retard.
A 999 call later, and checking the grey CCTV, the policeman says he reckons he knows where our stuff is, and they set off to a well known local toe-rags house, where they find the shit with several 2U servers, a spool of ethernet cable and they're busy trying to log into my locked laptop, which was running OpenSuse and totally baffled them.
The police seize everything, I can't get the job done, I call the boss and tell him, and basically tell the security guy to not let a soul past him until they show the ID's for the company.
Monday morning and I'm at the old site, showing people how to unplug their machines for transit to the new site, there's a fleet of vans to help now... You know, gotta move the leather chairs of the directors and the coffee machine properly, but the servers... ner just chuck them in the back of the car!
Needless to say, the ethernet all worked for everyone, replacement servers were easy enough to sort, and restore from the SAN, which was all fine and in place. The smashed server was harder to explain, and I chose not to, and let everyone assume the thieves had dropped it, whomever dropped it, it was well insured, and the replacement was the first of the then new Dell Poweredge's with the Itanium 2 in them!
But, I soon left this role, the company was sold to a larger parent and stripped of assets.
The drunk, he had my eternal ire after doing such a good job initially I found him not so appealing.
The local toe-rag, got 3 months, all the assets took, IIRC seven weeks to be returned to us, and some bright spark had used aluminium dust to take finger prints off of them... JOYEOUS!
No comments:
Post a Comment