Story Time - Fantastic Rack Mount Mistakes - Part 33 and 1/3

It has been a very long time since my last story time, and so this one comes to you with an unknown number, I am however well beyond any possible NDA or secrets for this one, so I can finally share (it's been 20 years yesterday).

I had a job in a data center, nothing too special, but I loved it.  The loud room, the air conditioning, the rack upon rack of machine, I absolutely loved it.  We're talking 2002 here, these were the 2U high style of machine I use a lot today, the odd 4U or 8U JBOD box too, loved them all.

I had all my wiring terminated and was beautifully dressed, I wasn't in charge of the whole place, just one isle, but I took my time whenever there was an issue to run new cable bundles beautifully dressed along the cabinet runners into each cabinet top, seven cabinets to a row, two face to face on the cold row and then the back of these into the hot, I had a hot row run by a lovely guy called Jerod next to me, he was French I think, lovely guy (shout out to Jerod if you're reading!).

The other side I had the work bench, a coveted space, between the hot and cold isles were thick plastic strip drapes, like you'd get in a meat packing plant.

We would patrol our isle, identifying any issues or disks showing physically bad, schedule them in for late night fix ups and then to into the communal office where we each hot desked.

Each of us had a laptop and would sit and connect into the monitoring suite remotely, with a big screen we connected via VGA to our laptops and dozens of spare keyboards we could quite often spend the night shifts doing nothing but playing games; Medieval Total War was the order of the day for me at the time; some of you know of my origins in the "game industry" take me back to hacking about and modifying MTW (Sorry Tom, whom I now work with, for fishing about breaking your baby).

Anyway, about three months into this job another tech on the day shift came to me about Isle 4, the far end of the room, he was getting drop outs at intervals during the day, but none at night, he was trying to figure out what might be causing it.

We took a look at his logs and sure enough on his disks he had 250-700ms delays for seek times on his platters, he also had a higher than average fault rate on disks, apparently he was also rebuilding raid arrays a lot.

He asked us in the night team to watch over the kit live; which we duely did.... Quiet as a mouse, nothing not an issue ever in about three months of monitoring.

Meanwhile in day light he was tearing his hair out; I've got to admit to genuinely not remembering his name, but he had long greasy black hair and was from Northampton which he told me more than his name... "Hello I'm <forgetable>, I live in Northampton".

But forgettable as he was I can't forget the day he handed that isle over to me.

Stumped as to what was going on, and frankly pulling rank, he shuffled Jerod down to Isle 3, himself into Isle 1 (my isle) and all new machines were coming into isle 2, which mr senior with disk failures was going to be setting up.

This really irked me, my isle was by far the best presented, when sales wanted to show new accounts around they showed them my isle and they all laughed over the name of the cabinets; we couldn't name the machines, most of them belongs to customers, but we could sticker and name the cabinets themselves, so they all got Bond Theme names.... Jaws, Octopussy, Moneypenny, Q and M all had simple servers, processors more or less and my two JBOD cabinets were Sean and Roger.  Stencilled on in lovely white enamel paint.

So I was hoofed off to isle 4, dark, at the back, no work bench... and immediately set about making it my own, I redressed the cable bundles (which annoyed Mr Senior, though he loved my already done ones, my dressing his was seen as an overt declaration of war).  It's so simple too, cable tie around five, cable tie through the middle to break them into three and two sets then pull.... Today you can get cable combs and I find them so sexy.

Anyway, dressed to impress what was not impressing was the fault rate, and instead of it being the accepted norm; as it had been for Mr Senior for months; it was now a big deal because I was in charge of the isle.

I set about trying to work it out, I could not figure it out.

At night, when I was there it was all fine, the day guys saw nothing; they pretty much just tended the machines remotely, no maintenance happened between 7am and 7pm so it was up to the night guys like me to swap disks, rebuild arrays, fix cables and install new machines.

So after two weeks with the log I got showing all these faults, like what was causing a disk which had worked all night and most of the day suddenly to show 500+ milliseconds seek!  It made no sense.

Anyway, 2002 was coming to a close, December 21st rolled around and I offered to cover a couple of the guys shifts, they had kids it was nearly Christmas and we worked Christmas Eve and half Christmas Day, one guy on on guy off for 6 hours at a time those two days, I took two shifts back to back.

Alone in the office, I watched the monitors and played games.

No spikes.... What gives, no spikes on the DAY I am actually here.

Hold on, I'm here, but no-one else is.

Could it be human interference?  I checked electrical circuits for noise, I checked lights, I checked if the air conditioning was affected outside and in.  I checked everything, no signs, no peaks, no noise.

HANG ON.  NO NOISE!

Spinning disks can be affected by loud noises, specifically by vibrations, I'd seen this demonstrated when doing my Compaq certification training.

There was no noise, could it just be vibration?

We were on the first floor (for anyone in the US, this is the floor above the one on the ground) so we are one storey up.  There was the main reception below center of the room, to the right where isle 1 is would be a hall way void with offices leading off it, under isle 4 would be a toilet and a changing room.

I went down, put the shower on, came back up.... Nope.

Then I took my laptop with me, no wifi, but I could plug it into various office ethernet ports around the place as no-one was in.

There I am Christmas Day, banging doors, flicking lights, flushing loos and then pouncing on my laptop to see if it affected any of the disk activity I was artificially running up.

Nothing.

Defeated I logged my time, handed over to Jerod and went to have my Christmas dinner.  I had 12 hours to think of something for Boxing day.

I walked into Mr Senior asking why there was CCTV of me "Dashing about with my laptop in random offices".

I just said I was trying to check light circuits for his disk issues; he made it abundantly clear they were my disk issues and left me to it.

Boxing day, the sales had started, the building was at the corner of a large commercial estate, there being a newly built Ikea across the road as the crowds rolled in and their stock levels fell they would be due a delivery soon.

I still poked around the office and the eureka moment came on the 27th December 2002.  For a large lorry was rumbling past the office, I had to wait for it to pass, a massive blue IKEA clad lorry.  I went into the office, logged into the monitoring and sure enough there was a trace of a large disk issue.  Times 4 minutes before; when that lorry went past!

I didn't hear it, I didn't feel it... But had the disks?

I waited and watched when a few lorries passed during the fairly quiet week between Christmas and New Years, nearly every heavy lorry going back resulted in some affect on the disks, vibration was being carried into the ground and I guess up through the building.  I will be honest, I could not tell.  But every spike I saw was timed with a lorry.  In fact I soon let Jerod in on my idea and plan to fix it and so he watched and I monitored and when he came in I could tell him when a lorry has passed!

I was not about to shout about this to Mr Senior, instead I set about fixing it.

I ordered a mat of 1 inch thick rubber, the stuff you mount washing machines on in your kitchen.  I already knew the racks pretty well, they were bolted at each foot, I'd need a torque wrench to unbolt them and I could use two of the hydraulic scissor platform lift trolleys we used to move machines about to lift the rack ever so slightly.

I didn't want to do this alone, so Jerod was roped in with the promise of a take away pizza from the glorious; but long gone; parlour we loved.

Mr Senior handed over to me and Jerod that night, it was not uncommon for two of us to be on at night, especially if there was work to do fitting something out.  It was normal to have two people when we were lifting machines too.  But Mr Senior would have been apoplectic if he'd know what us two kids were about to do.

I unscrewed the first rack from the floor, swept the dust out and used a wooden baton to bridge the lip to the metal of the scissor lift and I cranked it.... The cabinet moved, I nearly wet myself as it looked like it was going to topple, then Jerod jacked his side up on the other side and it came level, with an inch to spare I slipped in two of the pads.   He slipped in his two and at a shout over the loud AC we lowered.

I then used a phillips head screw driver to puncture the rubber and a knife to dig a bit out through each bolt hold and fastened the bolt back through, not too tight, just tight enough.

It was sweaty work in the hot isle as I was, but we did the first three of seven that night.

I went home and decided to pre cut the squares and use a drill to cut the middle out of them,the next night all seven were done.

Our disk fault rate when to zero.

That was my last time as an IT minion; I went back to programming soon after, and Mr Senior never was told what we did... His training should have told him.

Naming a new Storage Server?

 Server names?  In practice I've always been in places or projects where folks are bad at naming things and I am as guilty as the next person in that realm.  I've been on projects naming releases for obscure lord of the rings characters, for characters in a random fantasy story the lead developer had written, for birds, biomes, planets and stars.... Generally these are very niche, if you know the reference you're fine, and it brings a little character to your server.

The trouble arises when a name becomes attached to the history of that thing, the worst I never had was a project with a release named Elrond, where the release itself was re-released, so we ended up with Elrond-2, 3 and 4, then Elrond4RC1 for an update to that successful update, it was just a minefield.

In servers, the worst I ever managed was a box called "Diomede", which would often fail because of bad (read that as cheap) hardware, so the name became inextricably linked to bad performance and issues, despite being a really nice Tomcat & IBM Websphere host with a near 99% up-rate, but that 1% let it down because EVERYONE worked on that one box.

At home at the moment I have:

* Wellington

* Napoleon

* Elba

* Ney

* Waterloo

I'm sure you can tell the reference, but I've had about three versions of "Waterloo", I've also had two versions of "Ney" and at least five "Napoleons".  The names are getting mixed up, as I leave a home project and come back I'll go look up my notes, pull up that machine storage only to realise "oh this was on the OLD version of that name" which was long whisked away into the ethereal world of my replication server or just lost to time.

As such, I want to have ONE big disk server, and I really want to make it nice and neat.   I've already retired all my older machines, like this one:

But I do now have TWO cases of this specification, with the many internal drive bays:

In one of these I have a low-power Core i3 T class CPU, I forget the exact spec, but it's a low-power chip (35W) just enough power to run storage backups IMO, so this is where I'm going to start my quest.

Spinning rust drives are the challenge, I have four 4TB drives just sat idle, plus a host of smaller 1TB drives lying around, these need sorting out, mounting, checking and I then need a strategy.

I'm very likely to go with ZFS creating a mirror across three 1TB for critical storage and then some striped storage across the 4TB drives.  I then want this to make files available across the network, both to my 10GBit and my 1Gbit LAN's.... That's going to require a bit of work to install the former, but the latter should be available out of the box.

This starts to completely retire my 4Gbit fiber channel interconnects too.

So what to name this first stepping stone?.... Comments below:

Elba

Helena

Since it's about storage, I think my theme of names points to one of the islands which hosted Napoleon in exile... but lets see what you guys come up with.

How did I halve my Build speeds?... Oh

Deconstructing my build hive... As you may tell form earlier posts this month, I'm on a mission to sort out some of my equipment and either recycle it or whatever, and on this mission one of the first stops has been the older dual socket servers, which live in the garage on an ethernet spur at the moment.

And, I've taken them down, now I didn't really appreciate how much they lent to my build times, as my main app server (on which I coordinate the building of gcc projects with distcc) started to report 4x longer builds (on average) and I couldn't find a reason for it.

I looked at the app server, I looked at the jobs being thrown about and really there was no reason for the massive slow down...

Until I remembered something about my VDI set up, the app server used two sources for it's virtual disk images, one local and one was passed over to the containers on the garage servers, these images could be mounted over NFS but more regularly I had the garage servers sync them and mount locally from their local disks.

In migrating the build containers back locally to the app server, I'd moved the containers from their local loop mounted VDI's to ones hosted on the app server... From being mounted from 15,000 RPM SAS disks at about 12gb/s to a SATA 7,200 RPM disk at 6gb/s.

I'd essentially halved the disk speed of my builds, and totally forgot about setting this all up, now I did this years ago, it must be about two years since I set it up and it's been nearly a year since I moved house and started everything back up on the ethernet spur, which necessitated using the copy/sync and local mount option to get over the slow 10mbs ethernet in the garage.

My plan therefore is to get the new power supply, experiment with the supermicroo motherboard and then see if I can set up a new ZFS based file server on that SAS controller... Watch this space.

Personal PC Kit Stock Take - And First new PC Part Purchase

It begins.... A new PC build.

Yes, its that time, I simply need more power in my workstation on my desk.  I've been farming more and more work off to servers throughout the house, and I did a stock take... of all my kit... I've not done this since the house move, and indeed I documented the last stock take on these pages.

Workstation

Xeon X5670 (6 core, 12 thread) - 32GB RAM - Used all the time

Single Socket Servers

Xeon L5640 (6 core, 12 thread)- 8GB RAM - DistCC slave, Git Host & Fire Host - Used Rarely

Core 2 Duo E6750 (2 core, 2 thread) - 4GB RAM - General always on server (VPN Machine for Chaplain)

Dual Socket Servers

Xeon E5240 (2 core x 2) - 16GB RAM - Dist CC slave, Squid Host, Proxmox Host - Used Rarely

Xeon E5250 (2 core x 2) - 16GB RAM - Dist CC slave, Wow Classic Staging server - Used Rarely

Xeon E5430 (4 core x 2) - 32GB RAM - Dist CC, Dell BMC Test bed, ZFS mirror, Docker container host - Always On

Xeon E5430 (4 core x 2) - 32GB RAM - Xen Server Host.

Xeon E5340 (4 core x 1) - No RAM - Spares for other Dell Boxes

Xeon E5-2620 (6 core, 12 thread x 2) - 128GB RAM - DistCC build coordinator, main server - always on

It has to be said, nearly none of the server have been used regularly as I've just been so so busy.

Laptops

HP Omen (6 core, 12 thread) - 8GB RAM - Used Rarely

PC Specialist (4 core, 8 thread) - 8GB RAM - The Wifes machine, used as a network drive mirror

Lenovo ThinkPad E480 (4 core, 8 thread) - 16GB RAM - Main Laptop

What I decided is, a lot of this server infrastructure has got to go, and to do that the tasks they do - even in frequently have to be picked up by something, and I decided that as I usually spend my time in my new dedicated office I may as well have as much of that working here in the room with me.

The immediate points were to take all the work of the two main machines, the E5-2620 dual socket server (so 24 threads) and my workstation (12 threads), that's 36 threads of work to contend with doling out.

But, I immediately thought about keeping costs low.  The E5 class of Xeons have come down in price and have 10 cores and 20 threads, two of those maybe the way I decide to go... Forearmed with 40 threads, it's a case of getting a dual socket motherboard for my workstation, and since I game on the machine too, perhaps the slower L versions of that class wouldn't cut it... But research will be had.

128GB of RAM or more, seeing as these processors are last-gen that means DDR3 and specifically server EEC DDR3, which is fairly cheap.  And of course I have a whole bunch already in the house.  Unfortunately I have 128GB in 4GB sticks, but it's enough to get me up and testing x79 motherboards from aliexpress.

With the stock of parts I have on hand, it's therefore a the task of recasing things, making them quiet and having them bearably in the office with me.... I've been enamoured with the Corsair Obsidian range of cases for a while... But even before the case, power... 

My E5 based server is a Supermicro, but the motherboard within is actually just an EATX board, so with a spare PSU I could power it up in a new case maybe... And that's where I've decided to start... My first purchase therefore has been a new PSU.

https://www.amazon.co.uk/gp/product/B06ZZ3BYV7

This is platinum rated, that's highly highly efficient, it's all Japanese capacitors, it's workstation class and completely modular.

And you know what, it's only the third power supply I own, you may recall the Xmas Eve rush for a PC from a mate of mine, well I'm pretty sure that the problems there were the PSU in that machine... And I had no spare to offer, felt quite bad about that.  But now I will have spare, because my workstation will be shuffled down one notch into other devices.

The last focus I had on cases was cheapness...

The next one will be style & function.

My Home Network

My home network has been neglected, this is one of the problems of working in Technology, when you get home you are not going to be doing very much technology... Or maybe I'm just too interested in other things... Whatever the reason, neglect has set in.

Lets take a look at my home setup:

The network is broadly speaking split into two, the parts downstairs and the parts upstairs.

Downstairs is pretty much the Cable Modem box, a home hub set to Modem Only mode, connected to a Linksys router with some network attached storage for dumping files or media.

Upstairs is where the trouble begins, the critical path is to the left two upstairs items, my Xen server and especially my main Workstation, without these I can't work at all.  So thats two connections which I do not mess with, moving down we see the dotted yellow line, this is a single cable which lays on my desk that pays double duty to my powerful laptop or a raspberry Pi, so that's a spare I generally always need around.

We've used up four of our eight ports on the switch.  The final four are each going to the DRACs on my Dell Servers, which means they have no data connection.  If I want data to them (which is pretty much every time I use them) I either pull the wire from a neighbours DRAC, or I pull the cable from the IP Camera; it being the eighth and final port on the switch as it stands.

I have to ask what my options are... Well, I'm not able to change the cable trail from the lower to upper floor, so only one Cat6e cable there still, which means its not yet work my moving a box downstairs and dedicating it to pfSense & Squid.

I also don't have any rack space, so a racked switch with more ports is really a waste of money at the moment.

I took a look at another Netgear GC108 unmanaged switch, but I wondered about sub-net masking out some of the server stuff and thought for the few pounds different I'd go for a managed switch.

The desk then gets a new dedicated managed switch, and the more server stuff all stays together on the unmanged switch...

With the Dell servers immediately taking up six ports between their DRAC and data NICS I have two left, one for the Xen server and one to cross-connect with the managed switch.  When I come to rack these machines in a better manner, then I'll be able to co-locate the 8 port switch into whatever solution I have there without rewiring my main working station & desk!

This leaves me two ports free at the desk, in a unit which is light and small enough to screw to the underside of the unit (no way I could do that with a larger unit).

The real beauty for my needs here is the switch interlink, I can unplug just one wire and take away all my servers for re-positioning or remounting.  I could even take them all out of my immediate workspace now and hang them off of the WRT1900 downstairs (as it has two RJ45 ports spare).

Note to all the haters... On the topic of the Linksys WRT1900ACS, there is a lot of talk on the internet of it being flaky, unstable, crashing, resetting to factory defaults... Mine has been nothing but stable, like really stable, it's been reset once due to my needing to clear the cumulative six months of network map details and I think it locked up once with the WiFi not coming online.  This doesn't mean to go out there and buy one (though I did review it on Amazon, go take a look) but rather the unit I have was very good, and remains very good.

P.S. Neither Netgear nor Linksys sponsored my usage of their equipment, but you know, if you want to... Get in touch at the link near the Tip Jar!

From Ubuntu Server to Debian...

I have had a few issues with Ubuntu's 16.04 release, notably on the server side, when removing the physical network wire on one connection, it would stay connected on another, reconnecting the wire it'd not come back up.

I tried this with wireless connections, with physical RJ45 and virtual devices in various guises, none of them worked right for my needs.

And loathed as I am to change to anything else, I'm currently in the middle of setting up and using a new purely Debian based server... Watch this space....

Story Time : Fantastic Rack Mount Mistakes #3

Technically this is not a rack mounted mistake, but it was a mistake made by someone with a rack mounted unit.  A loooooooottttttttt of years ago, around the turn of the millennium in fact, I worked at a company where they used a Windows 2000 Server to collect a series of information, from memory, I think it ran Websphere and Jakarta Tomcat as middleware, connecting to a Microsoft SQL Server back end.

This server gave me nothing but trouble, I had to bring it down to publish new software to it, then bring it all back up again, because of some wierdness, and worst of all, I was only a user on the machine, I never saw it, was never in the same room as it, it was hosted in a data centre far away from me.

Remoting into it on late 1990's internet (because they'd not upgraded) was painful at 14kpbs, so much so that connecting to it from home was easier, where I had the first generation "Diamond Cable" (later NTL, later Virgin) broadband... at a blistering 512Kbps.

The main problem was, I needed someway to get the server to tell me it's IP, it could change, and the network guy at work used to handle this by crystal ball gazing, and copying the new IP into an e-mail and sending it to everyone.  Of course at home, I didn't get my work mail... Humpf.

The system however was all just Java, and after about 18 months I got the chance to move up the suckling chain of management within the project, and I immediately pointed out to the directors that we could drop all this Windows stuff, and the expensive licensing, by going to this new system called "Linux".

As a result I ended up at an open day presentation in the middle of Birmingham, with speakers from RedHat and Suse.  The talks were pointless, the guy from RedHat wore a T-shirt saying "Suse is shite", and the guy from Suse wore a suit and talked about professionalism; not very constructive, it all wanted to pull the admins in the room into the "are you a pro, or are you a hacker" division.  Neither talked about the distribution.

I therefore decided to learn about Linux myself, and I started a new server, initially just an old Pentium II machine with 32Mb of RAM, it was soon found to be a little bit better and a hell of a lot cheaper than the Windows Server, so I got a new shiny machine (my first commercial rack mounted server, I forget the model, I'm pretty sure it was a Compaq, as I'd seen the Proliant at my previous place of work and think I told the boss to get one).

This machine arrived, and it ran seamlessly pretty much from the get go, as had a couple of disk deaths in it's time, but there was no problem with it.

With the problem of it being in the remote location however, I had a trick, I created a script, like this:

ifconfig > config.txt

traceroute www.bbc.co.uk >> config.txt

cat config.txt | mail -s BootMail xelous@hotmail.co.uk

rm config.txt

The result?... I had it send me a mail whenever it reset, so I got the IP information, and a trace route listing to the outside world (I think the original place I used to trace route to was Altavista!  And it didn't go to hotmail, it went to my old NTL provided e-mail address.

What's the mistake here?... Well, that NTL e-mail address linked to my virgin e-mail address, which forwarded to my MSN address, which turned into Hotmail.

And... You may have guessed it... The other day, that very server was turned on somewhere, somehow... It had a network connection, was powered on, and it sent me a mail!

It was in Worcestershire, I think, and I could still SSH into it!

I changed the daily greeting to tell them they're server is insecure, disabled the script which was e-mailing me, and this pretty much ends our story.

The failure?  Don't plug things into your network, into your rack, unless you know what they're trying and going to do.  EVER!

Story Time : Fantastic Rack Mount Mistakes #2

Nothing is going to compare to the first post I made in this little series, but this was the first server room moment I had had to deal with.

I started work at a location, not very far from where I live now, and there was a little server room, again an old NCR mini-computer but this time it was running a custom cut of SCO-Unix, this machine was important, it handled everything for the whole midlands operation of this company.

And it was my first day...

There was a rack with kit on the right, and a rack with kit on the left, and you walked down the gap in the two... This was particularly bad, as it left you no room to move in there.

I was being shown around by the previous guy, whom was leaving, and I was really nervous... (Jim if you're out there, I hope you remember this).

So, we'd just concluded the tour of the server cupboard when I noted a black fleck in the air flow around me, another and another.  "Jim, there's black stuff".

Before I had the words out my mouth we got an awful smell of burning, everything was still on though.  All the network switches, the minicomputer, the UPS's the PC's running MSMail... Everything was still on, even the CRT monitors....

Something was certainly going wrong through, Jim stepped to the top of the room, leaving me in the middle isle, and he slid down the back of the rack and took a look, suddenly his arms were up and he's pulling the cables from a CRT monitor, and he yanks it down, it's clearly burning!

I pull the side door open and he skittles this thing down the corridor where it comes to a halt, happy it's not in the server room and not actually burning up we turn back to see what mess has been done.

The air has already been filtered so the soot removed, everything looked fine.  Except, unspotted by myself (I was only a few hours into this role) there was a little warning light on one of the UPS's, warning that it was on battery only.... I missed it, Jim missed it... Don however didn't miss it, as suddenly at the main door Don skidded into view, and nearly body tackled the pair of us inside the room, as he was down on his hip pulling at the mini computer.

"There's no power!"

The minicomputer was still on, but pressing a key on a terminal it was reporting it was busy shutting down.  The UPS was warning it had low power.... The company was offline.

Everything stopped... from ray role to purchasing, stopped.  Over 200 people, stopped.  COSTLY!

What had happened was as Jim had gone around the back, he'd stood on a cable, the mess of cables to be honest, behind the rack.

And one of them was the female to male coupler for the UPS to the wall, the plug was still in the wall, disappearing into the mess of cable, and then a cable emerging and plugged into the UPS, so it looked fine... and the UPS was connected to the minicomputer, with nice screwed in connectors... Hard to remove.

But this coupler was just laying there in the morass, and it was hard to see, and it was not screwed together at all... And it had come loose.

Needless to say, Jim was never remembered for saving the room from the fire, but remembered for unplugging the minicomputer.

Story Time : Fantastic Rack Mount Mistakes #1

This is the first in a little series, which I'm afraid will have no nice pictures, but trust me, these are all true and if not first hand, at least second hand accounts from trusted sources...

You may or may not have seen a server room in your time, if not then they're basically rooms, generally locked, with at least air conditioning, generally a fire extinguisher nearby and all your company server assets inside.

When I started out in business the server room was a large cupboard with an air conditioner and the 386 based NCR mini-computer inside.  It also had set of aluminium shelves, but it had no rack mounting, everything stood on the floor.

This was true for me, for a few years, the IBM AS400's we moved onto were huge machines on the floor, aside from the colour you felt like you were walking into the set of "Flight of the Navigator"

However, I soon moved over to Compaq Proliant machines and these were rack mounted... I walked into the server room, and took a look at the rack, only to be greeted by the UPS battery pack being at the top of the rack... Yes, the batteries, full of acid, were at the top...

I came out and went to speak to the nice chap in charge, pointing out, as gently as I could, that the batteries may leak, or if there was a fire could be ruptured, and it was not unknown for batteries to get hot swell & burst...

He stared at me, blinking, "What's wrong with that, they're in a case".

And he was damn right, they were... a Plastic case, this particular battery pack had no metal sheeting, no casing other made of metal, just metal corner brackets, it was essentially plastic...And full of acid.

In story number 2, you will hear about the server room fire I'd dealt with the previous year.  But this chap didn't seem to register a problem with having the battery pack at the top of the rack.

I did however lodge my complaint in the service book, and went about my work.  Months passed, and I didn't often go to this site, we moved all the other company assets to rack mounted ethernet switches, a few smaller rack mounted Pentium II based machines and all new desktops.

When one day, a user came to my office door and said they could not get onto the central server, checking my desktop serial terminal sure enough nothing.  Checking a physical VT100 nothing... Oh no.

I call the other site, no answer...

I get a secretary to start calling them every five minutes and I start to check back through our logs, it seems from our end the link was active at 6:03am about two hours before business would ramp up during the day, and the last items were a set of data from the design department at a Derby based location.

So I call Derby, asking "Did your upload complete at 6am?" a little bit of silence and I finally get the right chap on the line, he'd come in very early to get a jump on the day and to the best of my recall he said "No, it suddenly stopped and I could not get back on".

I cut him off as the secretary suddenly pages me (yes, rocking the pager) and I pick up my internal phone to hear I've got the guy from the server site on the line, he's coughing... "What happened?" is my instant question.

"Halon was active when I arrived"...

This is pretty bad, the server room was protected by an automatic halon fire extinguisher, if it detects smoke or heat there is a 10 second alert, the door closes and halon is released... 

Remember the lab scene in Terminator 2?...

A lot like that!... Staff in the room, or entering, need to use a breathing apparatus, which we'd all been trained on.

This guy was short of breath because he'd been in the halon room.

He says that when he arrived (at 7am) the room was in close down, the building fire alarm was going and a fire engine was already present... The security staff had let them in and they'd used breathing kit to enter the server room, to find....  And this was the best he could describe:

"A smouldering brown mess of plastic encasing the top of the Proliant server, stuck in all the front vents and solid into the quick release drive catches, all the network wires above are only copper and the battery pack is black"

I dive in a company car and drive up there, I arrive as the fire brigade are happy it's electrically isolated, that the halon is clear, we've got a guy from the fire suppression system coming to strut his stuff, but in the mean time the company is offline.

So, I set up a 10Mbit ethernet cable from the main input multiplexer, set up a backup system on an AST dual core machine (literally, it was a PC with two 486 CPU sockets) and 16mb of ram... power!

This was to be the tide over machine, we cycled the nightly backup onto that and at around noon, people were working, if extremely slowly.  That machine was our stop gap for the next 48 hours.

So, now to the server room, initially the most obvious problem (because the rack had a glass door closed) was the ceiling and floor... The ceiling tiles were seemingly not flame proof... DOH.... They'd melted at least and there were droplets of melted whatever they were all over the place, blown around by the AC.

The AC itself had valiantly kept fanning the flames, so it had black soot marks neatly blown all down the wall.

And the floor directly around the cabinet base was charred, they were fibre carpet tiles on concrete, but had smouldered...

Our fire plan had fired, and the halon tanks were empty, both had fired, despite perhaps this only needing one... The vent out had worked and we'd used up one of our internal breathing kits, so two spares, one inside one outside.

The room is also electrically dead, nothing is on inside.  We had redundant feeds in from sites, one inside the server room and the one outside down the corridor which the AST machine was now connected to.

We're using torches, so I opened the front door of the server rack, and it's just black, the glass is coated in black, the once cream coloured Compaqs are black, the silver rack arms are black, going up the patch panel above is black but you can see the wires look a lot thinner, reaching up they're charred, the plastic has sloughed off of them and later when I have a step ladder I see it's mostly pooled onto the top of the first computer, but it's also seeped into and set within the vents on top and spilled over into the front of the machine.

Above this is the crispy remains of the UPS, there is a hole about 2 by 3 inches around 8 inches from the left of the case bottom, and I can see into the UPS... It's a charred vision of hell.

We take this UPS out of the rack, as it is the ONLY thing in there that had been put in by the local chap, the ONLY things not on the service list, the ONLY thing I had had an issue with.... And we pop the lid, one battery had gone, catastrophically, it had melted acid out, the batteries lay on their side and so the corrosive acid had eaten the rubber off the outside of the wires inside, shorted and an electrical fire had ensued.

The battery acid had then, xenomorph style, eaten out of the bottom of the case and dripped onto the ethernet patch cables, they'd shorted, got hot, dripped more, the acid had also cut through the plastic on the power cables, more electrical shorts... And the gates of hell had opened swallowing the nice working machine.

The clean up was epic, we lifted the servers out to the cleaner area of the IT room, we cleaned the outside and had a pair of Compaq chaps come and clean them... I think they were called Mark and Jeff, and they did an excellent job.

I ordered new cable and thousands of RJ45 connectors and a crimper, setting about rewiring the room, in a better manner.

And I set the guy in charge of the site, Mr "It'll do" to taking out all the racking, jet washing it, putting it back in, redoing the power lines with a guy from the engineering department and I also had him remove all the carpet and the ceiling tiles, and find a supplier for a proper fireproof ceiling.

I also checked with the fire extinguisher guy, and he reckons that we were lucky the room was an hour outside of working hours, because the roof was just ceiling tiles and should never had been fitted with this strange fire system... Despite his being from the same company!... Basically most all the halon had exited via the ceiling tiles!

Two twenty hour days after all this I had one clean Compaq server, one clean rack and enough ethernet connections to put one patch panel and the server back online, this let our users stop moaning about how slow things were.

The other Compaq, the one closest the mess, was going off to be re-cased, the damage was so bad, but the unit itself worked.

We also now had a new fire safe, as the original had proven to not be smoke proof!

And I had a new certification on the walls being fire proof.

It took three months to recover from this, my end report to the owners, was that it'd cost about £8,500.  Not a bad deal considering the whole rebuild of the whole room would have been £40K+

I'd instigated better carpet, better ceiling, better procedure, and I had the brand new branded, supported, service contracted UPS in the bottom of the rack.

The final wash down, was the UPS which had failed, was known to be bad, three weeks before it'd gone up, the site manager had had it out and replaced two of the four battery cells within with a pair he'd bought cheap, he'd done this himself, I was not even aware, and he'd had the bottle to charge the company twice for the batteries, pocketing the difference.

Embezzlement, bad attitude, bad procedures and simply his considering me below him, resulted in his dismissal soon after.

He was the first person I enjoyed firing, he was the first person I'd actually had to actively fire from a role for incompetence, and he was the last person I let dictate to me where kit in a rack sat.

All for the placement of a UPS in a rack...

(Paul, Andy, Max if you recognise this story, you know you were a big part of that clear up, but you were in London, Leicester and Morocco... I was up to my arm pits in that mess!)

Dell 2950 Server - Fan Modification - Part 2

For the start of this silencing journey, go to part 1.

http://megalomaniacbore.blogspot.co.uk/2016/05/dell-2950-server-fan-modification-part-1.html

More details to follow in Part 3! (WHICH YOU CAN NOW SEE IN FULL!)

Software details: http://martin-denizet.com/silence-dell-poweredge-2950/

Blog Update: Mid-Session

Yeah, so what do I mean by mid-session?

Well, I'm midway through an awful lot of little projects, and right now (well about 5 hours ago) I actually just got home from a holiday (vacation to our American cousins)... So, lets just go over what I'm working on, what I'm planning for you guys, and when I plan to do it, saving time slips.

First, the server fan project, the fans have been modded with new resistors, they all still work, and the server boots.  And it a LOT LOT quieter, however, the BMC does start to moan after a while and it ramps the fans up to 100% screaming again when they drop below 2250 RPM.  So, I'm still working on software patching that...

Second, I have an introduction for you guys for the servers, covering all the little bits of kit I found in them, including memory installation, RAID controller configuration & installation, DRAC set up, and even re-working an Intel Dual Gigabit 8x PCIe.  Well I say re-working, I took the heat-sink off and put new shiny thermal paste on it to help it run cooler with the slower fans... Anyway, all that is coming up.

Third, what am I actually going to be doing with the servers?  Yes I have two new things to announce, watch this space.

Fourth, again a server item, racking them... I'm planning on building my own mini-coffee table style rack from some ikea items... However, this is a very low priority and depends on Pay-day coming around.

Last Server Item, it the new drives and RAM upgrades I've order, yes a whooping £20 has been spent!  However, I'm reaching out to several ebay and Amazon traders looking for partners to bring you new and exciting content as well as giving them reviews for their kit... So, watch this space for more kit being reviewed in my unique style.  To those vendors I've sent missives to... Get back to be folks, first come first served... I've got a Server CPU provider already negotiating... I'm missing RAM and cable providers... (hence why I've bought the first round of items for review & use).

Sixth, this is a software project, one of my most popular post series is my old, original, "Write your own CPU" series, which I think is pretty crap, I never finished it, never went beyond the basic add & sum op codes.  The later 4004 tutorial is not as popular (despite it being vastly superior), so to help my viewers see more of my blog and sites I'm going to be making a video series about writing your own virtual CPU.

I've got a bunch of real Z80 chips lying around, and some LED's, and a breadboard, so I may very well make a simple Z80 stepping/tester board and then we can write something similar in C++.

Seventh, Java code, the java series I had gets a fair view YouTube views, but I never see those viewers come through to here, so we need to figure out someway to get them java folks to read more than one line of text at a time!

So that whole series needs a tidy up, and the game itself there needs some new loving.

Eighth, I have some more Socket 775 modifications to work on, these are very very low on my list of projects, as I bought them at exactly the same time as the new servers came in.  And since the servers work, and are dual socket quad core Xeons, they are far more performance ready.

Very finally, I need to sort out the work-area, I have a new lovely digital controlled soldering iron, I have the Z80 project to solder, and three different arduino powered projects for the father-in-laws DCC train-set hobby to work on, so all those are coming this way, the first will be to mount the arc welding effect lighting into a single board solution.  Then I have an RGB effect lighting board to build to emulate a glowing set of fires for him.  Then finally a 5v DC motor project to power with PWM and emulate a set of coal-mine head stocks turning.

Dell 2950 Server - Fan Modification - Part 1

This is the first in a series I'm putting together, not to tutor, but to chronicle my altering the Dell 2950 servers I've just received to quieten them down.

The fans can be seen with the orange plastic handles.

I negotiated for the servers from a recycling shop, and got two for £80.  When the first two arrived, I was very happy with the look of them, one is 6 x 3.5" bays, one has 9 x 2.5" bays, but other than that they look in really good shape.

Inside they're clean, not perfect, but good.  Unfortunately someone not in the know has "set them up for shipping", so I have a bent front cheek on one unit, and then the memory in both is wrong, one has a single stick of 2GB RAM... and the other has 3 sticks of 512MB... First off, the servers ONLY take matched pairs, they complain without at least a matched pair... So from these sticks I can only manage to get 1GB as 2x512MB installed at the moment.

And then the big problem, was that one of the machines only has a single CPU installed, I only selected these as they were to come with two quad core CPU's...

But the vendor very kindly sent a replacement machine, unfortunately this machine has been dropped, so the rear of the case is very mangled.  But, never mind, that third machine is going to become a donor for spare parts on the other two!

So, what is the fan mod?  Well, I've read three different tutorials, taken a look at my unit, and watches lots of videos on the topic of quietening Dell server fans in general, but the 2950 specifically.

It is a two stage mod, the first is hardware, to fit a resistor to the red (power) cable on the four main input fans within the case.  This makes them run more slowly, and hence more quietly.

The problem with this is that the RPM meter then records a far lower rate, and so the system monitor/BIOS starts to complain that it has a defective fan, or four!...

This leads to the second stage of the modification, which is to modify the BIOS to stop this, by lowering the internally stored alert rates.

To prepare for the first step I removed my fans, and set about soldering the new resistors into place, I've started out with 47ohm 0.5W resistors, which are pretty high, some people have said they've had issues with fitting such a high resistance, in that the fan got way too slow.  So I have both 27 and 39 ohm resistors winging their way to me.

Once all four fans were soldered, I wrapped the fixes in insulation tape (I have no shrink wrap)... Refitting them of course left them running a lot slower, but the machine was not happy.

Therefore I'll move onto the next step, which is to fix the BIOS... See you all soon, to report how I get along.

Original Hand Wand Style Iron, which exploded

Oh, and during this step, I had the rubbish £10 soldering iron I was using explode (again), so I've gone and splashed out on a £50 digital one from Maplin, look out for a review of that soon!  (It soldered pretty well for a novice/hobbyist like me!)

Nice new Solder Station

Checking the work for continuity and then wrapping insulation.