Story Time : Fantastic Rack Mount Mistakes #2

Nothing is going to compare to the first post I made in this little series, but this was the first server room moment I had had to deal with.

I started work at a location, not very far from where I live now, and there was a little server room, again an old NCR mini-computer but this time it was running a custom cut of SCO-Unix, this machine was important, it handled everything for the whole midlands operation of this company.

And it was my first day...

There was a rack with kit on the right, and a rack with kit on the left, and you walked down the gap in the two... This was particularly bad, as it left you no room to move in there.

I was being shown around by the previous guy, whom was leaving, and I was really nervous... (Jim if you're out there, I hope you remember this).

So, we'd just concluded the tour of the server cupboard when I noted a black fleck in the air flow around me, another and another.  "Jim, there's black stuff".

Before I had the words out my mouth we got an awful smell of burning, everything was still on though.  All the network switches, the minicomputer, the UPS's the PC's running MSMail... Everything was still on, even the CRT monitors....

Something was certainly going wrong through, Jim stepped to the top of the room, leaving me in the middle isle, and he slid down the back of the rack and took a look, suddenly his arms were up and he's pulling the cables from a CRT monitor, and he yanks it down, it's clearly burning!

I pull the side door open and he skittles this thing down the corridor where it comes to a halt, happy it's not in the server room and not actually burning up we turn back to see what mess has been done.

The air has already been filtered so the soot removed, everything looked fine.  Except, unspotted by myself (I was only a few hours into this role) there was a little warning light on one of the UPS's, warning that it was on battery only.... I missed it, Jim missed it... Don however didn't miss it, as suddenly at the main door Don skidded into view, and nearly body tackled the pair of us inside the room, as he was down on his hip pulling at the mini computer.

"There's no power!"

The minicomputer was still on, but pressing a key on a terminal it was reporting it was busy shutting down.  The UPS was warning it had low power.... The company was offline.

Everything stopped... from ray role to purchasing, stopped.  Over 200 people, stopped.  COSTLY!

What had happened was as Jim had gone around the back, he'd stood on a cable, the mess of cables to be honest, behind the rack.

And one of them was the female to male coupler for the UPS to the wall, the plug was still in the wall, disappearing into the mess of cable, and then a cable emerging and plugged into the UPS, so it looked fine... and the UPS was connected to the minicomputer, with nice screwed in connectors... Hard to remove.

But this coupler was just laying there in the morass, and it was hard to see, and it was not screwed together at all... And it had come loose.

Needless to say, Jim was never remembered for saving the room from the fire, but remembered for unplugging the minicomputer.

Burning up, or shutting down, the Mini-Computer

I've decided today to tell a story, its a true story, and it bares some interesting information and history.  You see, I used to work for a company called "Claremont Garments", at the factory in Selton, and I worked in an IT support Role... I took over the role from a guy called Jim... The last time I saw Jim he fell out of a doorway near drunk when I picked him up for the annual departmental curry... But anyway, I was a student at the time... Studying Software Engineering.

The company ran an NCR brand mini-computer, running serial lines through multiplexors out to the distant sites, using dedicated comms lines.  Meaning in the far off factories there were green screen terminals showing the computer screen.

At the time it was the sunset of the mini-computer era, I don't think very many companies still run in that fashion, though some, like the one I work for now, do still have powerful servers running systems which are really just old software which should have a terminal, but which have been given a GUI.

But at the time, I was introduced to this NCR, and its AST backup, machine I was told it had to have an ALWAYS UP rule, I was shown how to make a call on the very expensive support contract, then told the push button code for the door, and shown inside the cave of technological delights.

It was all pretty old stuff, a wire wrack, with monitors, keyboards, UPS's and other stuff, there was networking switches running down the right of the room, and computers down the left.

But as I'm being shown what each computer does, I notice that in the stream of air from the air conditioner there's black flecks... Soot... I look at the unit, its a nice newish clean looking thing... Where's this black coming from... I glance down, nothing, glance up... There are three monitors on the top of this wrack of machines, the right hand one is on, but looks fuzzy, and the black particles are floating out of it... The ceiling tile is slightly discoloured with heat... That bastard thing is on FIRE!

Well, I shift back, and point it out, Jim to his credit dives in to his elbows and grabs this thing down... As I reach around his arms and unplug it... But now, he's holding a burning monitor over his head, and is stood in the server room... There are two doors to this room, I sling the one to the right open and Jim hurls this monitor over hand into the corridor, where it just sits black specks floating and all...

However, turning back around the chat hired to write code for the mini-computer, Don, is skidding into the other doorway and lands on the floor full length staring at the NCR... "What have you done?"  he cries "It's gone off"...

I look as puzzled as Jim, who retraces his steps... Whilst valiantly saving the server room from fire, he'd stood on a pile of wires, one of them was a coil of power lead, it was the lead from the UPS to the NCR... The NCR had lost power, instantly...

OH SHIT... Don and Jim reboot the machine, something I take care to learn about, but ironically in my whole time there, never had to do.

But then I take a look at the wire Jim "Pulled" out... it has screw fixings, it should have been screwed into the UPS and the NCR, but it wasn't... The person responsible for it not being screwed in, forever after, ridiculed Jim for unplugging the machine... But Jim had stood a good 4 feet to the side of the NCR itself... this chain of catastrophe should never had happened.

However, it did, and instead of Jim being remembered for his efforts to save the place burning up, he was always jibed and remembered for unplugging the NCR.