Showing posts with label IBM. Show all posts
Showing posts with label IBM. Show all posts

Monday, 4 December 2017

Great Rack Mount Mistakes #5

It has been a while since I've brought to you the tales of woe from my past... But this one isn't a tale of woe for myself, it was some other poor bugger who had to suffer, though I was involved.

After my first in-depth IT related job, I got into looking after some big systems, and I mean so Big they could have starred Tom Hanks... The last one of which ended, officially in early 2001, this was my looking after an IBM AS400 machine.

It had several terminals hooked into it, many suited analysts (as the non-programmers were called) regally sipped coffee and generated reports from it, there were also several ASCII Serial wireless hand-held terminals for roaming about the site with, all pretty cool.  I however was not involved in any of this, my job was to look after the PC's on the site and keep the AS400 fed with back-up tapes.

One of the PC's however took me within a solar breath of the chorona of glory that was working with the BIG IRON, and this was a little IBM PC, running OS/2 which was used actually boot the AS400, the mechanism escapes me, the details I've long forgotten, I remember it using OS/2, a terminal and a fancy script.

As I said this role ended for me in 2001, and I whisked my way off to work for a little software shop in Alcester, Warwickshhire (where I know I was a lazy pain in the arse - sorry about that lads - I grew up later, honest!).  Anyway, 23rd December 2001, I had a call at my parents home, a chap asking for me by name.  They handed out my personal mobile (Rocking the Nokia 3310 on Genie Mobile).

Well, this guy didn't let up, Christmas Eve, I'm mid-way through watching the Muppet Christmas Carol for the fifth time that day, and I finally look at the phone, and the seventeen texts to call the head-office of my prior employer.

Which I do, and awake a security guard whom had less hold of English than my dog, and the dog's Greek...

After a mixed conversation, I got through to a chap who was clearly in a server room, you could hear the noise behind him, I love that noise.

As he's talking to me however, the noise disappeared, dead silent...

"Did you just leave the server room?"

His reply.. "No, it just shuts off, it never completes its boot".

He was talking about the AS400, of which I knew nothing, they had very expensive IBM support for it on the way, but they were very worried, and wanted me to take a look, as everything had to be up for the Boxing Day sales - this was my time not in a manufacturing world, but in a retail world - the pressure was real, the target was live.

On the offer of a very nice cash sum, I jumped in my 206 and drove down to the offices, waved through security I signed in, and took a look around my old stomping ground, things had changed since I was last there, the partitioning wall to the server room had been removed, where the analysts sat was now occupied by a modern style series of server rack positions, they were in the process of moving everything to a set of Dell Power Edge 2U servers, with A/C, a hot isle and a cold, some decent kit.

Turning around there was the great big AS400, jet black, with a rectangular base rounded at one end.  And the machine perched on this raised platform.



This was the machine which would not booting... The problem?

Well, that paritioning wall, which had been removed... "How did you remove this wall?"

"Oh" he said "we had them put plastic sheets double lined from the ceiling to floor, took down the stud walling, and clean up, we never had to turn the AS400 off"

"Fabulous" I noted his pride "so the silver racks which were here, the shelving with parts and the little PC sat about here"  I intimated the corner just below waist height.

"All that was removed, with the wall, just old junk parts and pieces"

"Okay" I look around "So where did you remove it all to?"

"A skip" he shrugged "About three months ago"


"Aha" I nodded "I know your problem, the AS400 is coming up into advisory mode and awaiting the start signals from the script host, I'm going to guess you don't run AS400 elsewhere, you're winding down to the new servers?"


"Yeah" he was quite flustered "I know nothing about this hunk of junk, I just need it to work"

"Then you need to find a PC running OS/2 before morning, and restore from one of the back-ups I used to take onto tape, if you can"

He went whiter than my best linen on wash day... "OS2 PC?  Why?"

"Because the machine which sat here, was the boot master for the main shell into the Analysts layer, it sat here" indicating the wall again "whoever unplugged and threw it in a skip should really have looked a the holistic picture, it was a very important little machine, which is why we had two of them and spare parts on those shelves, and why it also got backed up to tape when it was installed or updates performed on it"

Silence filled the gap.

"I'll take that cash and get back to my Christmas pudding".




Friday, 9 September 2016

Story Time : Fantastic Rack Mount Mistakes #3

Technically this is not a rack mounted mistake, but it was a mistake made by someone with a rack mounted unit.  A loooooooottttttttt of years ago, around the turn of the millennium in fact, I worked at a company where they used a Windows 2000 Server to collect a series of information, from memory, I think it ran Websphere and Jakarta Tomcat as middleware, connecting to a Microsoft SQL Server back end.

This server gave me nothing but trouble, I had to bring it down to publish new software to it, then bring it all back up again, because of some wierdness, and worst of all, I was only a user on the machine, I never saw it, was never in the same room as it, it was hosted in a data centre far away from me.

Remoting into it on late 1990's internet (because they'd not upgraded) was painful at 14kpbs, so much so that connecting to it from home was easier, where I had the first generation "Diamond Cable" (later NTL, later Virgin) broadband... at a blistering 512Kbps.

The main problem was, I needed someway to get the server to tell me it's IP, it could change, and the network guy at work used to handle this by crystal ball gazing, and copying the new IP into an e-mail and sending it to everyone.  Of course at home, I didn't get my work mail... Humpf.

The system however was all just Java, and after about 18 months I got the chance to move up the suckling chain of management within the project, and I immediately pointed out to the directors that we could drop all this Windows stuff, and the expensive licensing, by going to this new system called "Linux".

As a result I ended up at an open day presentation in the middle of Birmingham, with speakers from RedHat and Suse.  The talks were pointless, the guy from RedHat wore a T-shirt saying "Suse is shite", and the guy from Suse wore a suit and talked about professionalism; not very constructive, it all wanted to pull the admins in the room into the "are you a pro, or are you a hacker" division.  Neither talked about the distribution.

I therefore decided to learn about Linux myself, and I started a new server, initially just an old Pentium II machine with 32Mb of RAM, it was soon found to be a little bit better and a hell of a lot cheaper than the Windows Server, so I got a new shiny machine (my first commercial rack mounted server, I forget the model, I'm pretty sure it was a Compaq, as I'd seen the Proliant at my previous place of work and think I told the boss to get one).

This machine arrived, and it ran seamlessly pretty much from the get go, as had a couple of disk deaths in it's time, but there was no problem with it.

With the problem of it being in the remote location however, I had a trick, I created a script, like this:

ifconfig > config.txt
traceroute www.bbc.co.uk >> config.txt
cat config.txt | mail -s BootMail xelous@hotmail.co.uk
rm config.txt

The result?... I had it send me a mail whenever it reset, so I got the IP information, and a trace route listing to the outside world (I think the original place I used to trace route to was Altavista!  And it didn't go to hotmail, it went to my old NTL provided e-mail address.

What's the mistake here?... Well, that NTL e-mail address linked to my virgin e-mail address, which forwarded to my MSN address, which turned into Hotmail.

And... You may have guessed it... The other day, that very server was turned on somewhere, somehow... It had a network connection, was powered on, and it sent me a mail!

It was in Worcestershire, I think, and I could still SSH into it!

I changed the daily greeting to tell them they're server is insecure, disabled the script which was e-mailing me, and this pretty much ends our story.

The failure?  Don't plug things into your network, into your rack, unless you know what they're trying and going to do.  EVER!