Great Rack Mount Mistakes #6

A long time coming, here's another story from my days long past, this one takes me to my very first serious role in an IT department, I was however just the dogs body.  The company ran many old PC's (which I actually was around to see mostly be updated to nice Compaq Pentium III's) and they had a couple of high spec Silicon Graphics workstations in the design department.

The main manufacturing control and purchasing system, as well as payroll and a bunch of other services ran on a dual 386 based mini computer, which had a custom cut of ScoUnix and a bunch of bespoke C programs comprising the actual system stack, this was accessed by a whole host or Gandalf multiplexers combining the serial connections down from a hundred or so Wyse brand terminals (I wish I'd have nabbed one of those before I left).

Anyway, it was time for this back end stack to be updated, and so a pair of Compaq Proliant servers were brought in, these were dual Pentium III class with a dedicated storage unit and a large; and importantly heavy; UPS unit.

The problem?  The IT manager I worked for (Hi Dave) didn't get on at all well with the manager at the co-location this unit was to be installed in.  Therefore in a dual effort to maintain any vestige of control and avoid the guy he didn't like, my boss ordered all the equipment to be delivered to our office... In central Nottinghamshire.... Yet its final destination once configured was to be outside Peterlee in the North East, near Newcastle.

So after around a day of setting up the equipment and (as far as I recall) three days solid compiling time - yes it took that long - the system was ready to go.

However, no-one had kept the boxes, yes it was all out of the box spread on a floor and then hand hauled over to a fire-exit and precariously piled into the back of a Hyundai estate.

Yes, that's how tens of thousands upon thousands of pounds worth of top notch equipment (for 1998) made it's precarious way 120 miles, bouncing and jostling all the way.

At the time I never questioned this, I was a lowly minion, I would of course council against such a move ever again, the installation of the physical equipment should have been done at the remote site, and they definitely should have kept the boxes and packaging in full!

Story Time : Fantastic Rack Mount Mistakes #3

Technically this is not a rack mounted mistake, but it was a mistake made by someone with a rack mounted unit.  A loooooooottttttttt of years ago, around the turn of the millennium in fact, I worked at a company where they used a Windows 2000 Server to collect a series of information, from memory, I think it ran Websphere and Jakarta Tomcat as middleware, connecting to a Microsoft SQL Server back end.

This server gave me nothing but trouble, I had to bring it down to publish new software to it, then bring it all back up again, because of some wierdness, and worst of all, I was only a user on the machine, I never saw it, was never in the same room as it, it was hosted in a data centre far away from me.

Remoting into it on late 1990's internet (because they'd not upgraded) was painful at 14kpbs, so much so that connecting to it from home was easier, where I had the first generation "Diamond Cable" (later NTL, later Virgin) broadband... at a blistering 512Kbps.

The main problem was, I needed someway to get the server to tell me it's IP, it could change, and the network guy at work used to handle this by crystal ball gazing, and copying the new IP into an e-mail and sending it to everyone.  Of course at home, I didn't get my work mail... Humpf.

The system however was all just Java, and after about 18 months I got the chance to move up the suckling chain of management within the project, and I immediately pointed out to the directors that we could drop all this Windows stuff, and the expensive licensing, by going to this new system called "Linux".

As a result I ended up at an open day presentation in the middle of Birmingham, with speakers from RedHat and Suse.  The talks were pointless, the guy from RedHat wore a T-shirt saying "Suse is shite", and the guy from Suse wore a suit and talked about professionalism; not very constructive, it all wanted to pull the admins in the room into the "are you a pro, or are you a hacker" division.  Neither talked about the distribution.

I therefore decided to learn about Linux myself, and I started a new server, initially just an old Pentium II machine with 32Mb of RAM, it was soon found to be a little bit better and a hell of a lot cheaper than the Windows Server, so I got a new shiny machine (my first commercial rack mounted server, I forget the model, I'm pretty sure it was a Compaq, as I'd seen the Proliant at my previous place of work and think I told the boss to get one).

This machine arrived, and it ran seamlessly pretty much from the get go, as had a couple of disk deaths in it's time, but there was no problem with it.

With the problem of it being in the remote location however, I had a trick, I created a script, like this:

ifconfig > config.txt

traceroute www.bbc.co.uk >> config.txt

cat config.txt | mail -s BootMail xelous@hotmail.co.uk

rm config.txt

The result?... I had it send me a mail whenever it reset, so I got the IP information, and a trace route listing to the outside world (I think the original place I used to trace route to was Altavista!  And it didn't go to hotmail, it went to my old NTL provided e-mail address.

What's the mistake here?... Well, that NTL e-mail address linked to my virgin e-mail address, which forwarded to my MSN address, which turned into Hotmail.

And... You may have guessed it... The other day, that very server was turned on somewhere, somehow... It had a network connection, was powered on, and it sent me a mail!

It was in Worcestershire, I think, and I could still SSH into it!

I changed the daily greeting to tell them they're server is insecure, disabled the script which was e-mailing me, and this pretty much ends our story.

The failure?  Don't plug things into your network, into your rack, unless you know what they're trying and going to do.  EVER!