Story Time - Fantastic Rack Mount Mistakes - Part 33 and 1/3

It has been a very long time since my last story time, and so this one comes to you with an unknown number, I am however well beyond any possible NDA or secrets for this one, so I can finally share (it's been 20 years yesterday).

I had a job in a data center, nothing too special, but I loved it.  The loud room, the air conditioning, the rack upon rack of machine, I absolutely loved it.  We're talking 2002 here, these were the 2U high style of machine I use a lot today, the odd 4U or 8U JBOD box too, loved them all.

I had all my wiring terminated and was beautifully dressed, I wasn't in charge of the whole place, just one isle, but I took my time whenever there was an issue to run new cable bundles beautifully dressed along the cabinet runners into each cabinet top, seven cabinets to a row, two face to face on the cold row and then the back of these into the hot, I had a hot row run by a lovely guy called Jerod next to me, he was French I think, lovely guy (shout out to Jerod if you're reading!).

The other side I had the work bench, a coveted space, between the hot and cold isles were thick plastic strip drapes, like you'd get in a meat packing plant.

We would patrol our isle, identifying any issues or disks showing physically bad, schedule them in for late night fix ups and then to into the communal office where we each hot desked.

Each of us had a laptop and would sit and connect into the monitoring suite remotely, with a big screen we connected via VGA to our laptops and dozens of spare keyboards we could quite often spend the night shifts doing nothing but playing games; Medieval Total War was the order of the day for me at the time; some of you know of my origins in the "game industry" take me back to hacking about and modifying MTW (Sorry Tom, whom I now work with, for fishing about breaking your baby).

Anyway, about three months into this job another tech on the day shift came to me about Isle 4, the far end of the room, he was getting drop outs at intervals during the day, but none at night, he was trying to figure out what might be causing it.

We took a look at his logs and sure enough on his disks he had 250-700ms delays for seek times on his platters, he also had a higher than average fault rate on disks, apparently he was also rebuilding raid arrays a lot.

He asked us in the night team to watch over the kit live; which we duely did.... Quiet as a mouse, nothing not an issue ever in about three months of monitoring.

Meanwhile in day light he was tearing his hair out; I've got to admit to genuinely not remembering his name, but he had long greasy black hair and was from Northampton which he told me more than his name... "Hello I'm <forgetable>, I live in Northampton".

But forgettable as he was I can't forget the day he handed that isle over to me.

Stumped as to what was going on, and frankly pulling rank, he shuffled Jerod down to Isle 3, himself into Isle 1 (my isle) and all new machines were coming into isle 2, which mr senior with disk failures was going to be setting up.

This really irked me, my isle was by far the best presented, when sales wanted to show new accounts around they showed them my isle and they all laughed over the name of the cabinets; we couldn't name the machines, most of them belongs to customers, but we could sticker and name the cabinets themselves, so they all got Bond Theme names.... Jaws, Octopussy, Moneypenny, Q and M all had simple servers, processors more or less and my two JBOD cabinets were Sean and Roger.  Stencilled on in lovely white enamel paint.

So I was hoofed off to isle 4, dark, at the back, no work bench... and immediately set about making it my own, I redressed the cable bundles (which annoyed Mr Senior, though he loved my already done ones, my dressing his was seen as an overt declaration of war).  It's so simple too, cable tie around five, cable tie through the middle to break them into three and two sets then pull.... Today you can get cable combs and I find them so sexy.

Anyway, dressed to impress what was not impressing was the fault rate, and instead of it being the accepted norm; as it had been for Mr Senior for months; it was now a big deal because I was in charge of the isle.

I set about trying to work it out, I could not figure it out.

At night, when I was there it was all fine, the day guys saw nothing; they pretty much just tended the machines remotely, no maintenance happened between 7am and 7pm so it was up to the night guys like me to swap disks, rebuild arrays, fix cables and install new machines.

So after two weeks with the log I got showing all these faults, like what was causing a disk which had worked all night and most of the day suddenly to show 500+ milliseconds seek!  It made no sense.

Anyway, 2002 was coming to a close, December 21st rolled around and I offered to cover a couple of the guys shifts, they had kids it was nearly Christmas and we worked Christmas Eve and half Christmas Day, one guy on on guy off for 6 hours at a time those two days, I took two shifts back to back.

Alone in the office, I watched the monitors and played games.

No spikes.... What gives, no spikes on the DAY I am actually here.

Hold on, I'm here, but no-one else is.

Could it be human interference?  I checked electrical circuits for noise, I checked lights, I checked if the air conditioning was affected outside and in.  I checked everything, no signs, no peaks, no noise.

HANG ON.  NO NOISE!

Spinning disks can be affected by loud noises, specifically by vibrations, I'd seen this demonstrated when doing my Compaq certification training.

There was no noise, could it just be vibration?

We were on the first floor (for anyone in the US, this is the floor above the one on the ground) so we are one storey up.  There was the main reception below center of the room, to the right where isle 1 is would be a hall way void with offices leading off it, under isle 4 would be a toilet and a changing room.

I went down, put the shower on, came back up.... Nope.

Then I took my laptop with me, no wifi, but I could plug it into various office ethernet ports around the place as no-one was in.

There I am Christmas Day, banging doors, flicking lights, flushing loos and then pouncing on my laptop to see if it affected any of the disk activity I was artificially running up.

Nothing.

Defeated I logged my time, handed over to Jerod and went to have my Christmas dinner.  I had 12 hours to think of something for Boxing day.

I walked into Mr Senior asking why there was CCTV of me "Dashing about with my laptop in random offices".

I just said I was trying to check light circuits for his disk issues; he made it abundantly clear they were my disk issues and left me to it.

Boxing day, the sales had started, the building was at the corner of a large commercial estate, there being a newly built Ikea across the road as the crowds rolled in and their stock levels fell they would be due a delivery soon.

I still poked around the office and the eureka moment came on the 27th December 2002.  For a large lorry was rumbling past the office, I had to wait for it to pass, a massive blue IKEA clad lorry.  I went into the office, logged into the monitoring and sure enough there was a trace of a large disk issue.  Times 4 minutes before; when that lorry went past!

I didn't hear it, I didn't feel it... But had the disks?

I waited and watched when a few lorries passed during the fairly quiet week between Christmas and New Years, nearly every heavy lorry going back resulted in some affect on the disks, vibration was being carried into the ground and I guess up through the building.  I will be honest, I could not tell.  But every spike I saw was timed with a lorry.  In fact I soon let Jerod in on my idea and plan to fix it and so he watched and I monitored and when he came in I could tell him when a lorry has passed!

I was not about to shout about this to Mr Senior, instead I set about fixing it.

I ordered a mat of 1 inch thick rubber, the stuff you mount washing machines on in your kitchen.  I already knew the racks pretty well, they were bolted at each foot, I'd need a torque wrench to unbolt them and I could use two of the hydraulic scissor platform lift trolleys we used to move machines about to lift the rack ever so slightly.

I didn't want to do this alone, so Jerod was roped in with the promise of a take away pizza from the glorious; but long gone; parlour we loved.

Mr Senior handed over to me and Jerod that night, it was not uncommon for two of us to be on at night, especially if there was work to do fitting something out.  It was normal to have two people when we were lifting machines too.  But Mr Senior would have been apoplectic if he'd know what us two kids were about to do.

I unscrewed the first rack from the floor, swept the dust out and used a wooden baton to bridge the lip to the metal of the scissor lift and I cranked it.... The cabinet moved, I nearly wet myself as it looked like it was going to topple, then Jerod jacked his side up on the other side and it came level, with an inch to spare I slipped in two of the pads.   He slipped in his two and at a shout over the loud AC we lowered.

I then used a phillips head screw driver to puncture the rubber and a knife to dig a bit out through each bolt hold and fastened the bolt back through, not too tight, just tight enough.

It was sweaty work in the hot isle as I was, but we did the first three of seven that night.

I went home and decided to pre cut the squares and use a drill to cut the middle out of them,the next night all seven were done.

Our disk fault rate when to zero.

That was my last time as an IT minion; I went back to programming soon after, and Mr Senior never was told what we did... His training should have told him.

Great Rack Mount Mistakes #7

Today's story in the annuls of problems in IT comes from a guest editor... Mr B.... And Mr B (no relation to anyone in other stories given monogram names) works as the sysadmin and developer for the whole set of systems with his employer; unfortunately this means "it's all his fault".

So what went wrong?  Well, over night the site had a power cut and though they have a nice server, they don't have a power back up, so that server went off.

The server is essentially a java host, specifically hosting Tomcat, and it reaches out to connect to a set of third party endpoints via a restful API.

You'd think no big deal, start up get running and keep running, except that third party don't force a disconnect upon a new edition of their interface API, if you're connected to version 1.0 then you can and they will happily leave you connected to version 1.0, even if they release interim updates, add new calls and quite what got Mr B today remove a call or two.

Your session ending, and then I presume all sessions of that old version, would free their server provisioning to de-allocate the old version.  But to force users to migrate upwards in the chain their published API (so think the end point here in whatever flavour you wish) declaration changes.  Such that you re-download it upon re-connection and that's your new flavour of the month API.

The problem?  It didn't work.

So Mr B had to set about debugging this on the fly, in a live environment, which was down.  And he went through the three stages of technological grief....

1) Denial:    "This is completely illogical, my code brings down their interface, which is the only thing we connect to, it must be right, they can't miss-match them, so this must be my side or the gods are against me".

2) Investigation:  "Read the logs, make a change, nothing seems to work, the gods are definitely snickering behind that cloud of steam now".

3) Realisation:   "If it's not me, and it's not the system here, it must be their side, the huge multi-billion international must have published their API spec with a mistake or miss-match.... click.... YOU FUCKERS!"

What was the actual problem?  Well, the third party published API was actually wrong, the downloaded specification still contained several calls which were removed, when the services Mr B had written came up they checked each end point and found several calls defined which did not respond and so his software, correctly, reported that the endpoint was offline.  They were, they didn't exist anymore.

His fix was to literally tell his stuff to ignore the multi-billion dollar international service providers API spec and to "download" a copy which he hosted locally, with his own edits to it.

Now, he's a tiny fish in a huge pond here, even if he reports this miss-match said multi-billion dollar international isn't going to hear him, and by the time he does they maybe several months down the line, and other folks may have spotted this problem.  He maybe listened to, but he essentially doubts his voice would be heard.

The problem of course being how to abate this issue in the future?  How to avoid this stress?  For at one point he did say "the company is done for", because literally everything was offline, all their services were down.... And of course everyone will blame the little guy doing all the IT, they won't think that the multi-billion behemoth entity could possibly publish a wonky API spec, most of those shouting at Mr B with mouths frothing wouldn't even know what he meant when he explained this to them...

The fact that he's identified this issue, resolved it, and everything is back up within two hours won't be remembered, the glass will remain half-empty, and so it'll only be remembered that on the 3rd September 2020 Mr B's IT suite went offline.

Ikea Server Rack is Finished

What kind of bloke gets to play with servers on his day off?  A damn lucky one my wife always tell me?.. Am I lucky?  Or do I have a plan?  I think I have a plan, this is a project I've been meaning to sort out for quite some time, now I have a week off of work, I'm knocking things out the park... Sooo......

Meet my new server wrack, so this is a pair of Ikea Lack Coffee tables, one with the feet fitted, and it's screwed down onto the top of another, with a pair of wheel which I also got at Ikea.

I've reinforced the leg uprights, and I'm busy fitting out the rear with a dedicated switch and power setup.

It's all on wheels so it can go anywhere in my office with relative ease.

And yes, these are my older Poweredge servers (from my fan/noise reduction videos).

Only the top unit is operational, with all the drives fitted and a full compliment of memory (64GB I think is in it).  Its socking a quad port gigabyte Ethernet at the moment but I have a whole box of fiber channel cards to go into the machines up here taking me up to 4Gbit per card (full duplex) and I plan to fit two cards in each machine so we should bond up to about 8gbits... We'll see, waiting on the cables to arrive from Hong Kong.

My other item is to get myself a dual Socket 1366 based server soon, just to play about with 6 core 12 thread CPU's (so up to 12 cores 24 threads) and perhaps in the new year I'll go up to the E5- series of Xeons to massive amounts of threads.

At the moment though, load balancing various things on my AWS instances and dropping to my own hosting is more my interests...

And a friend of mine has reminded me of a project he and I need undertake - watch this space.

Great Rack Mount Mistakes #6

A long time coming, here's another story from my days long past, this one takes me to my very first serious role in an IT department, I was however just the dogs body.  The company ran many old PC's (which I actually was around to see mostly be updated to nice Compaq Pentium III's) and they had a couple of high spec Silicon Graphics workstations in the design department.

The main manufacturing control and purchasing system, as well as payroll and a bunch of other services ran on a dual 386 based mini computer, which had a custom cut of ScoUnix and a bunch of bespoke C programs comprising the actual system stack, this was accessed by a whole host or Gandalf multiplexers combining the serial connections down from a hundred or so Wyse brand terminals (I wish I'd have nabbed one of those before I left).

Anyway, it was time for this back end stack to be updated, and so a pair of Compaq Proliant servers were brought in, these were dual Pentium III class with a dedicated storage unit and a large; and importantly heavy; UPS unit.

The problem?  The IT manager I worked for (Hi Dave) didn't get on at all well with the manager at the co-location this unit was to be installed in.  Therefore in a dual effort to maintain any vestige of control and avoid the guy he didn't like, my boss ordered all the equipment to be delivered to our office... In central Nottinghamshire.... Yet its final destination once configured was to be outside Peterlee in the North East, near Newcastle.

So after around a day of setting up the equipment and (as far as I recall) three days solid compiling time - yes it took that long - the system was ready to go.

However, no-one had kept the boxes, yes it was all out of the box spread on a floor and then hand hauled over to a fire-exit and precariously piled into the back of a Hyundai estate.

Yes, that's how tens of thousands upon thousands of pounds worth of top notch equipment (for 1998) made it's precarious way 120 miles, bouncing and jostling all the way.

At the time I never questioned this, I was a lowly minion, I would of course council against such a move ever again, the installation of the physical equipment should have been done at the remote site, and they definitely should have kept the boxes and packaging in full!

My Home Network

My home network has been neglected, this is one of the problems of working in Technology, when you get home you are not going to be doing very much technology... Or maybe I'm just too interested in other things... Whatever the reason, neglect has set in.

Lets take a look at my home setup:

The network is broadly speaking split into two, the parts downstairs and the parts upstairs.

Downstairs is pretty much the Cable Modem box, a home hub set to Modem Only mode, connected to a Linksys router with some network attached storage for dumping files or media.

Upstairs is where the trouble begins, the critical path is to the left two upstairs items, my Xen server and especially my main Workstation, without these I can't work at all.  So thats two connections which I do not mess with, moving down we see the dotted yellow line, this is a single cable which lays on my desk that pays double duty to my powerful laptop or a raspberry Pi, so that's a spare I generally always need around.

We've used up four of our eight ports on the switch.  The final four are each going to the DRACs on my Dell Servers, which means they have no data connection.  If I want data to them (which is pretty much every time I use them) I either pull the wire from a neighbours DRAC, or I pull the cable from the IP Camera; it being the eighth and final port on the switch as it stands.

I have to ask what my options are... Well, I'm not able to change the cable trail from the lower to upper floor, so only one Cat6e cable there still, which means its not yet work my moving a box downstairs and dedicating it to pfSense & Squid.

I also don't have any rack space, so a racked switch with more ports is really a waste of money at the moment.

I took a look at another Netgear GC108 unmanaged switch, but I wondered about sub-net masking out some of the server stuff and thought for the few pounds different I'd go for a managed switch.

The desk then gets a new dedicated managed switch, and the more server stuff all stays together on the unmanged switch...

With the Dell servers immediately taking up six ports between their DRAC and data NICS I have two left, one for the Xen server and one to cross-connect with the managed switch.  When I come to rack these machines in a better manner, then I'll be able to co-locate the 8 port switch into whatever solution I have there without rewiring my main working station & desk!

This leaves me two ports free at the desk, in a unit which is light and small enough to screw to the underside of the unit (no way I could do that with a larger unit).

The real beauty for my needs here is the switch interlink, I can unplug just one wire and take away all my servers for re-positioning or remounting.  I could even take them all out of my immediate workspace now and hang them off of the WRT1900 downstairs (as it has two RJ45 ports spare).

Note to all the haters... On the topic of the Linksys WRT1900ACS, there is a lot of talk on the internet of it being flaky, unstable, crashing, resetting to factory defaults... Mine has been nothing but stable, like really stable, it's been reset once due to my needing to clear the cumulative six months of network map details and I think it locked up once with the WiFi not coming online.  This doesn't mean to go out there and buy one (though I did review it on Amazon, go take a look) but rather the unit I have was very good, and remains very good.

P.S. Neither Netgear nor Linksys sponsored my usage of their equipment, but you know, if you want to... Get in touch at the link near the Tip Jar!

Story Time : Fantastic Rack Mount Mistakes #4

The final story in my little series, really revolves around the dread of all server room administrators... More nerve shredding than fire!  More nerve tingling than granting terminal access!.... We are of course talking about moving server room....

The plan was eeked out of management in literally, fifteen minutes of a meeting... We had three 6' racks all the wiring, all 2U servers in one rack with a 32 port switch at the top, nicely wired by good self (wishes I had a picture, but we were still using film for photo's back then, and it was too much effort).

The middle rack was a bit more the experimental kit, there was a 4U server in there, a couple of desktop cased machines stood directly on top of that unit - with plastic cutouts to stop hot air from the back feeding back to the front of the rack - yes, even with ghetto server layouts forced upon me, I did try to make the best of things and keep good order...

The third rack was telecomms units, it contained a bank of modems (yes, 64K modems!) another ethernet switch, an unmanaged switch too, and our SAN.

The SAN was a Dell EquiLogic; from memory I remember having 16 drives in it of 120Gb each, and two were hot spares...

The plan, from management, "unplug it all, load it into the two Nissan Primera Estate company cars, and move it from Derbyshire to Leicestershire"... 

That was it, "unplug and move"!

We had two days, over a weekend, to do this... It was Thursday afternoon... The clock was on.

Before you ask, I can't tell you much about why we had to move, nor whom I was working for, but let us just say, a week after the move the company changed name, and I changed employment soon after.

So, Thursday afternoon, I sent out e-mail to everyone, all staff, literally everyone, and said "The Servers are ALL going off, the Network is coming down, ALL IT services will be suspended at 3pm the following day".

Of course the whole place started to moan, so I promptly sent another mail.. "No seriously, ALL Services will be offline at 3pm, I'm serious!"  And I left the building to go see this new site... Two hours driving later, it's nearing 5pm Thursday and I arrive at a tin-shack looking warehouse like building, am let in by a man with a carrier bag and a cough.  I get the keys off of him to, and go check out the "IT Room" as he calls it.

It's a desk, with a single 15amp 240v rail supply and a bunch of cut off ethernet cables spilling out the wall in one corner.

There is no aircon, there is no patch panel, there is no racking... And it's on the first floor... (If you're from the US and reading this, here in the UK the floor against the ground is called "the ground floor", and the "first floor" is above that, it is up at least one flight of stairs... Yep, stairs... No lift (elevator).

Company credit card out, I head back to the office, and I order on next day (before 11am) delivery two RJ45 crimpers, two spools of ethernet cable, a pair of wall mountable 8U cabinets and a laptop (you know, because I thought I might need one) and a pair of walkie-talkies.

I then survey the kit I have to move, now, I can ignore the modems in the large, they unplug and lift out, but there's no telecomms due at the other end for a fortnight, so they'll go dark, they can be the last thing we move.  The switches in that right hand rack though therefore become free, and I can take them our now and start to set up the ethernet at the other end.

I have been assigned a minion from the shop floor to help, Friday morning comes and I await until 1pm that afternoon for my "before 11am delivery"... things are going so swimmingly already.  But I have had time to take the two desktop form factor servers offline, the 4U unit off line, the front blanking all out and disassemble the middle rack.  I've also got the switches from the right hand telecoms cabinet...

In our first transfer load we therefore have the rack, cables, UPS's, a 1U fold out terminal and crimpers in one car.  The two desktop servers and the 4U unit in the other.  We drive to the new site, unload the servers onto a trolley and lift them up stairs, placing them in a corner on the floor.

We then pull up the rack, assemble it, and fit a switch to the top of the cabinet and I send my helper off to survey the site, looking for any and all RJ45 sockets, and noting where they are, room by room and any notes on them.  I meanwhile start to splice new RJ45 connectors onto every one of the fifty or so ethernet cables laying on the floor, and I plug them into the switch.  I braid them all and cap them and wrap them nicely into the rack cable managements either side.

The switch has a useful line-test function, and my minion has the laptop!... So he can just plug a cable in at the other end and I see a light come on...  Over the radio I ask him to pick a socket, tell me if it's marked and to plug in... We do this room by room, and I move patches around to group rooms together, and I tell him a number to mark on the socket with a sharpie.

We've been at this mind numbing task for a couple of hours when I decide we need a break, and I go down to meet him... I'm a little aghast to see he's not written on the little window areas for the socket numbers, he's written them in letters as tall as the socket itself... *slaps forehead*.... It looks a mess, but it works, I'll have to come around in a weeks time with some acetone to clean it up, but it'll do... It'll do.

Fifty odd sockets later, we've got the whole building ethernet hooked into this switch.  The rack is powered and the UPS's are stable, reporting good, and the 4U server is up and fine from the laptop over a few of the sockets I've checked.

The two desktop servers survive the move too, and they're placed in the footing of the rack, rather then perched ontop.

Everything looks fab, and we've got this all done before 7pm on the Friday!  Things are looking GOOOODD!

I buy the chap helping me a pub meal, and he takes the company car home, I expect to see him Saturday morning.

Saturday morning, before switching down everything in the office, I check everyone is offline, and start to kill the servers down.  The SAN down, and pulling everything out of the rack, I'm stacking all the patch cables I'm taking all the units out and the car is pretty full.  Where is the other car?... 9am... 10am... No sign, I've got to call him.

He's at home, in bed... His wife describes him as, "Drunk as a lord"... He apparently didn't realise I needed him the next day, and had a skin full after he'd gone home.  He can't drive.

So, I'm flying solo... 

I drive the first load down to the new site, and basically throw it all out the back into the loading dock, and set off for the next load, I get the UPS's for this rack in the car and two of the 2U servers... and it's pretty much at load... Four more trips and I have all the servers at the new site, in the loading bay, and I'm taking the left hand rack down at the old place.

It won't come apart, and it's really a two man job... So, I give up, and I load all the bits I can up, and go to the new site, and decide to just stack the machines up off the floor but in a pile, rather then rack mounted, for a while.

So, machine by machine, I lug them up the stairs... It's heavy, it's hard, and I'm on the penultimate machine... When half way up the first flight of stairs I feel I'm about to loose my grip.

I try to catch a better hold, but it all happens at once, the machine is free, I'm toppling to my side as the weight takes me, and I'm going down the stairs on my side with my left finger in the thumb hook on the front of the server.

Lets just say, it was my left finger... And the server fell to my right.... Yeah, snappy snappy.

So, I have definitely broken my finger.  And I've definitely broken the server, as it bounced down about eight hard steps, the real flanges on the rack rails are all bent, the handles on the PSU units are smashed and the drives have shook loose...

Swearing and hobbling I crawl over to the machine, pop the lid and it's all fucked, just properly fucked.  The motherboard has popped up, everything is a mess.... That's an expensive mistake, but not-one I've ever admitted to, until now.

My finger however, needs attention, so I lock up and take a trip to A&E... After three hours they basically say it's a bad break, and put my hand in a mini-cast across the back of my hand and down my finger itself its all fixed.  I can use my wrist.

I return to the site, and luckily for me, there are no more servers to be lifted upstairs... Because someone has hiked open the loading dock roller door and pinched the one remaining server which was sat there.... They've also taken the laptop from upstairs, the two desktop profile servers and a bunch of other stuff...

When I go talk to the "security" guy, the old geeza with the bag from the previous day, he's still carrying his bag... I ask if he saw anything, "yeah saw one of your lads moving your computer things"... I explain he wasn't one of our lads... "I thought that funny, he didn't lift the roller, but crawled under it"... Total retard.

A 999 call later, and checking the grey CCTV, the policeman says he reckons he knows where our stuff is, and they set off to a well known local toe-rags house, where they find the shit with several 2U servers, a spool of ethernet cable and they're busy trying to log into my locked laptop, which was running OpenSuse and totally baffled them.

The police seize everything, I can't get the job done, I call the boss and tell him, and basically tell the security guy to not let a soul past him until they show the ID's for the company. 

Monday morning and I'm at the old site, showing people how to unplug their machines for transit to the new site, there's a fleet of vans to help now... You know, gotta move the leather chairs of the directors and the coffee machine properly, but the servers... ner just chuck them in the back of the car!

Needless to say, the ethernet all worked for everyone, replacement servers were easy enough to sort, and restore from the SAN, which was all fine and in place.  The smashed server was harder to explain, and I chose not to, and let everyone assume the thieves had dropped it, whomever dropped it, it was well insured, and the replacement was the first of the then new Dell Poweredge's with the Itanium 2 in them!

But, I soon left this role, the company was sold to a larger parent and stripped of assets.

The drunk, he had my eternal ire after doing such a good job initially I found him not so appealing.

The local toe-rag, got 3 months, all the assets took, IIRC seven weeks to be returned to us, and some bright spark had used aluminium dust to take finger prints off of them... JOYEOUS!