Wiley IT Manager Saving on Microsoft Licences

It is late 1998 and I observe one of the smartest PC acquisition steps ever; the company I work for is reequipping every one using a certain application to a new specification machine.  This means a major purchase and roll out for those of us in IT.

A little background first the company I work for has one customer, just one single customer, this may sound crazy but at the time it made perfect sense and the customer was a reassuring British stalwart of the high street the business was rock solid.  They were however also extremely protective of their brand and selling our whole product line exclusively they knew they could tell us how to do everything.

One of the key things they specified was a hard encryption model to their stock and control system software, we had to run the software they provided, we had to run is with "secure" physical dongles performing periodic security authorisation checks and we did all this on a specification of machine they laid down to us.

So it was in 1998 a new version of this software was hoving into view, the specification leapt from a mere 486 running 25hmz to a Pentium III running at or over 120mhz.  The RAM requirements went from 4MB and windows 3.11 for workgroups to the then brand new Windows 98.

So it was my boss (shout out to Dave) set about working out the best platform for this.

He had previously been in charge of the purchase of new server stack from Compaq and with a positive impression he turned to them.

With a little wrangling I believe he had a roll out of 30 machines, with three years support, for £890.  In terms today this is approximately £2000 a seat and was just for the machine a 17" monitor, keyboard, mouse and windows 98.  Nothing else.

Folks had to do other tasks on these machines, not just this customer software, therefore he set about buying Office.  Homogenising the previous smorgasbord array of different spreadsheet and word processing software variously in use.

Adding Office 97 SBE unfortunately pushed the machines up another £80 per seat, this included £19 off for bulk purchase, but it was a crazy price.

But then Dave taught me an extremely valuable lesson, to play the edge cases.

Could you get Office 97 for les than £80?  Yes, you could get it for £49.95 a seat.  But only if it was an upgrade.  Hmm, what could we count as an upgrade from?

Well, it turned out Office 97 could be an upgrade from MS Works 95.  At the time Works was my go to office package, I've never felt the ease and familiarity with office ever again since.  But works was canned by Microsoft; probably because of Wiley IT managers like Dave.

For a full new copy of Works would be had for just £12.95.

Doing a little mathematics, £12.95 + £49.95 is a mere £62.90.

The company was already duty bound to pay me as part of my regular services, so installing Works and installing Word over the top, taking hours to get through all the machine did result in quite a saving.  About £400 for the whole project; meaning Dave was well under budget and everything worked as intended.

It did however leave one literally huge problem; for the next working year our already tiny IT office was overrun with these dozens of double boxes of Works and Office upgrade, just in case Redmond came knocking asking about licenses.

Great Rack Mount Mistakes #6

A long time coming, here's another story from my days long past, this one takes me to my very first serious role in an IT department, I was however just the dogs body.  The company ran many old PC's (which I actually was around to see mostly be updated to nice Compaq Pentium III's) and they had a couple of high spec Silicon Graphics workstations in the design department.

The main manufacturing control and purchasing system, as well as payroll and a bunch of other services ran on a dual 386 based mini computer, which had a custom cut of ScoUnix and a bunch of bespoke C programs comprising the actual system stack, this was accessed by a whole host or Gandalf multiplexers combining the serial connections down from a hundred or so Wyse brand terminals (I wish I'd have nabbed one of those before I left).

Anyway, it was time for this back end stack to be updated, and so a pair of Compaq Proliant servers were brought in, these were dual Pentium III class with a dedicated storage unit and a large; and importantly heavy; UPS unit.

The problem?  The IT manager I worked for (Hi Dave) didn't get on at all well with the manager at the co-location this unit was to be installed in.  Therefore in a dual effort to maintain any vestige of control and avoid the guy he didn't like, my boss ordered all the equipment to be delivered to our office... In central Nottinghamshire.... Yet its final destination once configured was to be outside Peterlee in the North East, near Newcastle.

So after around a day of setting up the equipment and (as far as I recall) three days solid compiling time - yes it took that long - the system was ready to go.

However, no-one had kept the boxes, yes it was all out of the box spread on a floor and then hand hauled over to a fire-exit and precariously piled into the back of a Hyundai estate.

Yes, that's how tens of thousands upon thousands of pounds worth of top notch equipment (for 1998) made it's precarious way 120 miles, bouncing and jostling all the way.

At the time I never questioned this, I was a lowly minion, I would of course council against such a move ever again, the installation of the physical equipment should have been done at the remote site, and they definitely should have kept the boxes and packaging in full!

Story Time : Fantastic Rack Mount Mistakes #3

Technically this is not a rack mounted mistake, but it was a mistake made by someone with a rack mounted unit.  A loooooooottttttttt of years ago, around the turn of the millennium in fact, I worked at a company where they used a Windows 2000 Server to collect a series of information, from memory, I think it ran Websphere and Jakarta Tomcat as middleware, connecting to a Microsoft SQL Server back end.

This server gave me nothing but trouble, I had to bring it down to publish new software to it, then bring it all back up again, because of some wierdness, and worst of all, I was only a user on the machine, I never saw it, was never in the same room as it, it was hosted in a data centre far away from me.

Remoting into it on late 1990's internet (because they'd not upgraded) was painful at 14kpbs, so much so that connecting to it from home was easier, where I had the first generation "Diamond Cable" (later NTL, later Virgin) broadband... at a blistering 512Kbps.

The main problem was, I needed someway to get the server to tell me it's IP, it could change, and the network guy at work used to handle this by crystal ball gazing, and copying the new IP into an e-mail and sending it to everyone.  Of course at home, I didn't get my work mail... Humpf.

The system however was all just Java, and after about 18 months I got the chance to move up the suckling chain of management within the project, and I immediately pointed out to the directors that we could drop all this Windows stuff, and the expensive licensing, by going to this new system called "Linux".

As a result I ended up at an open day presentation in the middle of Birmingham, with speakers from RedHat and Suse.  The talks were pointless, the guy from RedHat wore a T-shirt saying "Suse is shite", and the guy from Suse wore a suit and talked about professionalism; not very constructive, it all wanted to pull the admins in the room into the "are you a pro, or are you a hacker" division.  Neither talked about the distribution.

I therefore decided to learn about Linux myself, and I started a new server, initially just an old Pentium II machine with 32Mb of RAM, it was soon found to be a little bit better and a hell of a lot cheaper than the Windows Server, so I got a new shiny machine (my first commercial rack mounted server, I forget the model, I'm pretty sure it was a Compaq, as I'd seen the Proliant at my previous place of work and think I told the boss to get one).

This machine arrived, and it ran seamlessly pretty much from the get go, as had a couple of disk deaths in it's time, but there was no problem with it.

With the problem of it being in the remote location however, I had a trick, I created a script, like this:

ifconfig > config.txt

traceroute www.bbc.co.uk >> config.txt

cat config.txt | mail -s BootMail xelous@hotmail.co.uk

rm config.txt

The result?... I had it send me a mail whenever it reset, so I got the IP information, and a trace route listing to the outside world (I think the original place I used to trace route to was Altavista!  And it didn't go to hotmail, it went to my old NTL provided e-mail address.

What's the mistake here?... Well, that NTL e-mail address linked to my virgin e-mail address, which forwarded to my MSN address, which turned into Hotmail.

And... You may have guessed it... The other day, that very server was turned on somewhere, somehow... It had a network connection, was powered on, and it sent me a mail!

It was in Worcestershire, I think, and I could still SSH into it!

I changed the daily greeting to tell them they're server is insecure, disabled the script which was e-mailing me, and this pretty much ends our story.

The failure?  Don't plug things into your network, into your rack, unless you know what they're trying and going to do.  EVER!

Story Time : Fantastic Rack Mount Mistakes #2

Nothing is going to compare to the first post I made in this little series, but this was the first server room moment I had had to deal with.

I started work at a location, not very far from where I live now, and there was a little server room, again an old NCR mini-computer but this time it was running a custom cut of SCO-Unix, this machine was important, it handled everything for the whole midlands operation of this company.

And it was my first day...

There was a rack with kit on the right, and a rack with kit on the left, and you walked down the gap in the two... This was particularly bad, as it left you no room to move in there.

I was being shown around by the previous guy, whom was leaving, and I was really nervous... (Jim if you're out there, I hope you remember this).

So, we'd just concluded the tour of the server cupboard when I noted a black fleck in the air flow around me, another and another.  "Jim, there's black stuff".

Before I had the words out my mouth we got an awful smell of burning, everything was still on though.  All the network switches, the minicomputer, the UPS's the PC's running MSMail... Everything was still on, even the CRT monitors....

Something was certainly going wrong through, Jim stepped to the top of the room, leaving me in the middle isle, and he slid down the back of the rack and took a look, suddenly his arms were up and he's pulling the cables from a CRT monitor, and he yanks it down, it's clearly burning!

I pull the side door open and he skittles this thing down the corridor where it comes to a halt, happy it's not in the server room and not actually burning up we turn back to see what mess has been done.

The air has already been filtered so the soot removed, everything looked fine.  Except, unspotted by myself (I was only a few hours into this role) there was a little warning light on one of the UPS's, warning that it was on battery only.... I missed it, Jim missed it... Don however didn't miss it, as suddenly at the main door Don skidded into view, and nearly body tackled the pair of us inside the room, as he was down on his hip pulling at the mini computer.

"There's no power!"

The minicomputer was still on, but pressing a key on a terminal it was reporting it was busy shutting down.  The UPS was warning it had low power.... The company was offline.

Everything stopped... from ray role to purchasing, stopped.  Over 200 people, stopped.  COSTLY!

What had happened was as Jim had gone around the back, he'd stood on a cable, the mess of cables to be honest, behind the rack.

And one of them was the female to male coupler for the UPS to the wall, the plug was still in the wall, disappearing into the mess of cable, and then a cable emerging and plugged into the UPS, so it looked fine... and the UPS was connected to the minicomputer, with nice screwed in connectors... Hard to remove.

But this coupler was just laying there in the morass, and it was hard to see, and it was not screwed together at all... And it had come loose.

Needless to say, Jim was never remembered for saving the room from the fire, but remembered for unplugging the minicomputer.

Story Time : Fantastic Rack Mount Mistakes #1

This is the first in a little series, which I'm afraid will have no nice pictures, but trust me, these are all true and if not first hand, at least second hand accounts from trusted sources...

You may or may not have seen a server room in your time, if not then they're basically rooms, generally locked, with at least air conditioning, generally a fire extinguisher nearby and all your company server assets inside.

When I started out in business the server room was a large cupboard with an air conditioner and the 386 based NCR mini-computer inside.  It also had set of aluminium shelves, but it had no rack mounting, everything stood on the floor.

This was true for me, for a few years, the IBM AS400's we moved onto were huge machines on the floor, aside from the colour you felt like you were walking into the set of "Flight of the Navigator"

However, I soon moved over to Compaq Proliant machines and these were rack mounted... I walked into the server room, and took a look at the rack, only to be greeted by the UPS battery pack being at the top of the rack... Yes, the batteries, full of acid, were at the top...

I came out and went to speak to the nice chap in charge, pointing out, as gently as I could, that the batteries may leak, or if there was a fire could be ruptured, and it was not unknown for batteries to get hot swell & burst...

He stared at me, blinking, "What's wrong with that, they're in a case".

And he was damn right, they were... a Plastic case, this particular battery pack had no metal sheeting, no casing other made of metal, just metal corner brackets, it was essentially plastic...And full of acid.

In story number 2, you will hear about the server room fire I'd dealt with the previous year.  But this chap didn't seem to register a problem with having the battery pack at the top of the rack.

I did however lodge my complaint in the service book, and went about my work.  Months passed, and I didn't often go to this site, we moved all the other company assets to rack mounted ethernet switches, a few smaller rack mounted Pentium II based machines and all new desktops.

When one day, a user came to my office door and said they could not get onto the central server, checking my desktop serial terminal sure enough nothing.  Checking a physical VT100 nothing... Oh no.

I call the other site, no answer...

I get a secretary to start calling them every five minutes and I start to check back through our logs, it seems from our end the link was active at 6:03am about two hours before business would ramp up during the day, and the last items were a set of data from the design department at a Derby based location.

So I call Derby, asking "Did your upload complete at 6am?" a little bit of silence and I finally get the right chap on the line, he'd come in very early to get a jump on the day and to the best of my recall he said "No, it suddenly stopped and I could not get back on".

I cut him off as the secretary suddenly pages me (yes, rocking the pager) and I pick up my internal phone to hear I've got the guy from the server site on the line, he's coughing... "What happened?" is my instant question.

"Halon was active when I arrived"...

This is pretty bad, the server room was protected by an automatic halon fire extinguisher, if it detects smoke or heat there is a 10 second alert, the door closes and halon is released... 

Remember the lab scene in Terminator 2?...

A lot like that!... Staff in the room, or entering, need to use a breathing apparatus, which we'd all been trained on.

This guy was short of breath because he'd been in the halon room.

He says that when he arrived (at 7am) the room was in close down, the building fire alarm was going and a fire engine was already present... The security staff had let them in and they'd used breathing kit to enter the server room, to find....  And this was the best he could describe:

"A smouldering brown mess of plastic encasing the top of the Proliant server, stuck in all the front vents and solid into the quick release drive catches, all the network wires above are only copper and the battery pack is black"

I dive in a company car and drive up there, I arrive as the fire brigade are happy it's electrically isolated, that the halon is clear, we've got a guy from the fire suppression system coming to strut his stuff, but in the mean time the company is offline.

So, I set up a 10Mbit ethernet cable from the main input multiplexer, set up a backup system on an AST dual core machine (literally, it was a PC with two 486 CPU sockets) and 16mb of ram... power!

This was to be the tide over machine, we cycled the nightly backup onto that and at around noon, people were working, if extremely slowly.  That machine was our stop gap for the next 48 hours.

So, now to the server room, initially the most obvious problem (because the rack had a glass door closed) was the ceiling and floor... The ceiling tiles were seemingly not flame proof... DOH.... They'd melted at least and there were droplets of melted whatever they were all over the place, blown around by the AC.

The AC itself had valiantly kept fanning the flames, so it had black soot marks neatly blown all down the wall.

And the floor directly around the cabinet base was charred, they were fibre carpet tiles on concrete, but had smouldered...

Our fire plan had fired, and the halon tanks were empty, both had fired, despite perhaps this only needing one... The vent out had worked and we'd used up one of our internal breathing kits, so two spares, one inside one outside.

The room is also electrically dead, nothing is on inside.  We had redundant feeds in from sites, one inside the server room and the one outside down the corridor which the AST machine was now connected to.

We're using torches, so I opened the front door of the server rack, and it's just black, the glass is coated in black, the once cream coloured Compaqs are black, the silver rack arms are black, going up the patch panel above is black but you can see the wires look a lot thinner, reaching up they're charred, the plastic has sloughed off of them and later when I have a step ladder I see it's mostly pooled onto the top of the first computer, but it's also seeped into and set within the vents on top and spilled over into the front of the machine.

Above this is the crispy remains of the UPS, there is a hole about 2 by 3 inches around 8 inches from the left of the case bottom, and I can see into the UPS... It's a charred vision of hell.

We take this UPS out of the rack, as it is the ONLY thing in there that had been put in by the local chap, the ONLY things not on the service list, the ONLY thing I had had an issue with.... And we pop the lid, one battery had gone, catastrophically, it had melted acid out, the batteries lay on their side and so the corrosive acid had eaten the rubber off the outside of the wires inside, shorted and an electrical fire had ensued.

The battery acid had then, xenomorph style, eaten out of the bottom of the case and dripped onto the ethernet patch cables, they'd shorted, got hot, dripped more, the acid had also cut through the plastic on the power cables, more electrical shorts... And the gates of hell had opened swallowing the nice working machine.

The clean up was epic, we lifted the servers out to the cleaner area of the IT room, we cleaned the outside and had a pair of Compaq chaps come and clean them... I think they were called Mark and Jeff, and they did an excellent job.

I ordered new cable and thousands of RJ45 connectors and a crimper, setting about rewiring the room, in a better manner.

And I set the guy in charge of the site, Mr "It'll do" to taking out all the racking, jet washing it, putting it back in, redoing the power lines with a guy from the engineering department and I also had him remove all the carpet and the ceiling tiles, and find a supplier for a proper fireproof ceiling.

I also checked with the fire extinguisher guy, and he reckons that we were lucky the room was an hour outside of working hours, because the roof was just ceiling tiles and should never had been fitted with this strange fire system... Despite his being from the same company!... Basically most all the halon had exited via the ceiling tiles!

Two twenty hour days after all this I had one clean Compaq server, one clean rack and enough ethernet connections to put one patch panel and the server back online, this let our users stop moaning about how slow things were.

The other Compaq, the one closest the mess, was going off to be re-cased, the damage was so bad, but the unit itself worked.

We also now had a new fire safe, as the original had proven to not be smoke proof!

And I had a new certification on the walls being fire proof.

It took three months to recover from this, my end report to the owners, was that it'd cost about £8,500.  Not a bad deal considering the whole rebuild of the whole room would have been £40K+

I'd instigated better carpet, better ceiling, better procedure, and I had the brand new branded, supported, service contracted UPS in the bottom of the rack.

The final wash down, was the UPS which had failed, was known to be bad, three weeks before it'd gone up, the site manager had had it out and replaced two of the four battery cells within with a pair he'd bought cheap, he'd done this himself, I was not even aware, and he'd had the bottle to charge the company twice for the batteries, pocketing the difference.

Embezzlement, bad attitude, bad procedures and simply his considering me below him, resulted in his dismissal soon after.

He was the first person I enjoyed firing, he was the first person I'd actually had to actively fire from a role for incompetence, and he was the last person I let dictate to me where kit in a rack sat.

All for the placement of a UPS in a rack...

(Paul, Andy, Max if you recognise this story, you know you were a big part of that clear up, but you were in London, Leicester and Morocco... I was up to my arm pits in that mess!)