Story Time : Fantastic Rack Mount Mistakes #1

This is the first in a little series, which I'm afraid will have no nice pictures, but trust me, these are all true and if not first hand, at least second hand accounts from trusted sources...

You may or may not have seen a server room in your time, if not then they're basically rooms, generally locked, with at least air conditioning, generally a fire extinguisher nearby and all your company server assets inside.

When I started out in business the server room was a large cupboard with an air conditioner and the 386 based NCR mini-computer inside.  It also had set of aluminium shelves, but it had no rack mounting, everything stood on the floor.

This was true for me, for a few years, the IBM AS400's we moved onto were huge machines on the floor, aside from the colour you felt like you were walking into the set of "Flight of the Navigator"

However, I soon moved over to Compaq Proliant machines and these were rack mounted... I walked into the server room, and took a look at the rack, only to be greeted by the UPS battery pack being at the top of the rack... Yes, the batteries, full of acid, were at the top...

I came out and went to speak to the nice chap in charge, pointing out, as gently as I could, that the batteries may leak, or if there was a fire could be ruptured, and it was not unknown for batteries to get hot swell & burst...

He stared at me, blinking, "What's wrong with that, they're in a case".

And he was damn right, they were... a Plastic case, this particular battery pack had no metal sheeting, no casing other made of metal, just metal corner brackets, it was essentially plastic...And full of acid.

In story number 2, you will hear about the server room fire I'd dealt with the previous year.  But this chap didn't seem to register a problem with having the battery pack at the top of the rack.

I did however lodge my complaint in the service book, and went about my work.  Months passed, and I didn't often go to this site, we moved all the other company assets to rack mounted ethernet switches, a few smaller rack mounted Pentium II based machines and all new desktops.

When one day, a user came to my office door and said they could not get onto the central server, checking my desktop serial terminal sure enough nothing.  Checking a physical VT100 nothing... Oh no.

I call the other site, no answer...

I get a secretary to start calling them every five minutes and I start to check back through our logs, it seems from our end the link was active at 6:03am about two hours before business would ramp up during the day, and the last items were a set of data from the design department at a Derby based location.

So I call Derby, asking "Did your upload complete at 6am?" a little bit of silence and I finally get the right chap on the line, he'd come in very early to get a jump on the day and to the best of my recall he said "No, it suddenly stopped and I could not get back on".

I cut him off as the secretary suddenly pages me (yes, rocking the pager) and I pick up my internal phone to hear I've got the guy from the server site on the line, he's coughing... "What happened?" is my instant question.

"Halon was active when I arrived"...

This is pretty bad, the server room was protected by an automatic halon fire extinguisher, if it detects smoke or heat there is a 10 second alert, the door closes and halon is released... 

Remember the lab scene in Terminator 2?...

A lot like that!... Staff in the room, or entering, need to use a breathing apparatus, which we'd all been trained on.

This guy was short of breath because he'd been in the halon room.

He says that when he arrived (at 7am) the room was in close down, the building fire alarm was going and a fire engine was already present... The security staff had let them in and they'd used breathing kit to enter the server room, to find....  And this was the best he could describe:

"A smouldering brown mess of plastic encasing the top of the Proliant server, stuck in all the front vents and solid into the quick release drive catches, all the network wires above are only copper and the battery pack is black"

I dive in a company car and drive up there, I arrive as the fire brigade are happy it's electrically isolated, that the halon is clear, we've got a guy from the fire suppression system coming to strut his stuff, but in the mean time the company is offline.

So, I set up a 10Mbit ethernet cable from the main input multiplexer, set up a backup system on an AST dual core machine (literally, it was a PC with two 486 CPU sockets) and 16mb of ram... power!

This was to be the tide over machine, we cycled the nightly backup onto that and at around noon, people were working, if extremely slowly.  That machine was our stop gap for the next 48 hours.

So, now to the server room, initially the most obvious problem (because the rack had a glass door closed) was the ceiling and floor... The ceiling tiles were seemingly not flame proof... DOH.... They'd melted at least and there were droplets of melted whatever they were all over the place, blown around by the AC.

The AC itself had valiantly kept fanning the flames, so it had black soot marks neatly blown all down the wall.

And the floor directly around the cabinet base was charred, they were fibre carpet tiles on concrete, but had smouldered...

Our fire plan had fired, and the halon tanks were empty, both had fired, despite perhaps this only needing one... The vent out had worked and we'd used up one of our internal breathing kits, so two spares, one inside one outside.

The room is also electrically dead, nothing is on inside.  We had redundant feeds in from sites, one inside the server room and the one outside down the corridor which the AST machine was now connected to.

We're using torches, so I opened the front door of the server rack, and it's just black, the glass is coated in black, the once cream coloured Compaqs are black, the silver rack arms are black, going up the patch panel above is black but you can see the wires look a lot thinner, reaching up they're charred, the plastic has sloughed off of them and later when I have a step ladder I see it's mostly pooled onto the top of the first computer, but it's also seeped into and set within the vents on top and spilled over into the front of the machine.

Above this is the crispy remains of the UPS, there is a hole about 2 by 3 inches around 8 inches from the left of the case bottom, and I can see into the UPS... It's a charred vision of hell.

We take this UPS out of the rack, as it is the ONLY thing in there that had been put in by the local chap, the ONLY things not on the service list, the ONLY thing I had had an issue with.... And we pop the lid, one battery had gone, catastrophically, it had melted acid out, the batteries lay on their side and so the corrosive acid had eaten the rubber off the outside of the wires inside, shorted and an electrical fire had ensued.

The battery acid had then, xenomorph style, eaten out of the bottom of the case and dripped onto the ethernet patch cables, they'd shorted, got hot, dripped more, the acid had also cut through the plastic on the power cables, more electrical shorts... And the gates of hell had opened swallowing the nice working machine.

The clean up was epic, we lifted the servers out to the cleaner area of the IT room, we cleaned the outside and had a pair of Compaq chaps come and clean them... I think they were called Mark and Jeff, and they did an excellent job.

I ordered new cable and thousands of RJ45 connectors and a crimper, setting about rewiring the room, in a better manner.

And I set the guy in charge of the site, Mr "It'll do" to taking out all the racking, jet washing it, putting it back in, redoing the power lines with a guy from the engineering department and I also had him remove all the carpet and the ceiling tiles, and find a supplier for a proper fireproof ceiling.

I also checked with the fire extinguisher guy, and he reckons that we were lucky the room was an hour outside of working hours, because the roof was just ceiling tiles and should never had been fitted with this strange fire system... Despite his being from the same company!... Basically most all the halon had exited via the ceiling tiles!

Two twenty hour days after all this I had one clean Compaq server, one clean rack and enough ethernet connections to put one patch panel and the server back online, this let our users stop moaning about how slow things were.

The other Compaq, the one closest the mess, was going off to be re-cased, the damage was so bad, but the unit itself worked.

We also now had a new fire safe, as the original had proven to not be smoke proof!

And I had a new certification on the walls being fire proof.

It took three months to recover from this, my end report to the owners, was that it'd cost about £8,500.  Not a bad deal considering the whole rebuild of the whole room would have been £40K+

I'd instigated better carpet, better ceiling, better procedure, and I had the brand new branded, supported, service contracted UPS in the bottom of the rack.

The final wash down, was the UPS which had failed, was known to be bad, three weeks before it'd gone up, the site manager had had it out and replaced two of the four battery cells within with a pair he'd bought cheap, he'd done this himself, I was not even aware, and he'd had the bottle to charge the company twice for the batteries, pocketing the difference.

Embezzlement, bad attitude, bad procedures and simply his considering me below him, resulted in his dismissal soon after.

He was the first person I enjoyed firing, he was the first person I'd actually had to actively fire from a role for incompetence, and he was the last person I let dictate to me where kit in a rack sat.

All for the placement of a UPS in a rack...

(Paul, Andy, Max if you recognise this story, you know you were a big part of that clear up, but you were in London, Leicester and Morocco... I was up to my arm pits in that mess!)

WarThunder : BF109 - Stealth Damage Model Change

I would swear that there has been a stealth change to the damage model of the BF109... And a change at the fundamental level, take a look at these three screenshots....

They are three different fly outs...

Do you notice the common element?... yes, a wing tip missing, the spar exposed - but still present?.... and the aircraft can still be flown and controlled, but you are counted as shot down!... In the lower screen shot I actually held that E4 straight and level for five minutes and nearly managed to hold it til the end of the game!

But, this damage seems to come out of nowhere, and always against low-calibre fire!  It's almost like Gaijin have decided "Take a low cal hit in that box, and your wing tip falls off"...

There's no pattern to it, no reason, I've has it from behind, from head-on, from oblique angles, even from passing bombers!... BOOM no wing tip... 

And it's such a sudden change in and stark alteration of the damage model...

Grinding Halt

It was only a matter of time, but it seems my active push of late has finally led me to falling ill, yes I'm really run down, and the holiday season has let me relax enough to let a massively heavy chest attack me.  Nothing like as bad as the proper pneumonia I had earlier in the year, but enough to bring progress on all things house & decoration to a halt.

All I've managed game-wise is to extend my branch mining in Minecraft, and to gain access to the first real zero the A6M2 in WarThunder - yes by proper I mean the one without the fucking floats....

But I really need to get better now, I want, nay need, to finish the room here... I need the two big walls plastering and then the whole thing painting, and then I want to get down the carpet shop and pick out some new threads for me to plod about on...

And then I need to start the laminate flooring downstairs.

This later endeavour is already beset with problems, the foremost being, I don't have a power saw or jig to hold the planks and so it'll be wonky as wonky-McWonkerson.... I need to therefore invest in a mitre saw bench/clamp/something... And then I need to actually prepare the floors, the front room floor has a large bulge in it, near the doorway, which is not conducive to laying a flat laminate floor on top of... gah.

Man Lab - Plastering Planning to Finish the Alcove

I'm actually writing this on like the 18th, but there were lots of posts planned and scheduled, so you're getting it today - late as it were.

Tonight I'm going to try to get the left hand side done on the main wall, but I'm doing this in two steps because of my poor mixing so far... So, I'm going to get the skim bead up on the left of the main window - the blue squares...

Next I'm going to do one mix and try to get from the wall around the left most window, now I've already got skim bead around this window, and I've already done the inside of it... so its smooth, the wall above it however has that bulge mentioned before, so its going to be a shit.

 

Whilst that's having a moment to set I'm going to clean the bucket and get clean water... Then smooth the first left most skim out... and clean the wall corner up... And then its the second mix and I'm going to do the blank flat wall between the two windows.

 

Now this skim has to get to the radiator at the bottom, there's no gap at the top of the radiator below the sill, so that's as far as we can get along the bottom there, and then blend and taper off the skim at the top there...

Problems with this section is that the top left of the right hand (main) window has been rebuilt in bonding compound, so I need to scrape some of that away, also the whole area above the windows bulges.

But, once this skim is done the desk can go up.... And the plan is that it'll fit into the alcove I've already plastered... here's my MS Paint powered 3D rendering...

So the desk, and shelf want to go in like this:

The reason being... a) to get it done... b) the desk fixings can go up... c) the father-in-law coming xmas day can help cus the desk to size and fit the shelf.

Now, this whole plastering effort he's going to take the piss out of, but... hey I've seen the plastering done by someone else in our utility room... that's god awful plastering compared to mine, and the wife seems to like my efforts here.

Oh, I've also been trying to drop hints to people that I need a simple DIY skimming plastering course, there is one I'd go to in the local area, but it seems to be more of other details I don't need... like putting up dry lining board... and then another which seems exactly what I want is like 80 miles away... Both courses are expensive, over £180 for one and £225 the other... I'm in negotiation with the wife to go on one, but this room needs finishing before that, so... Wish me luck.

----------------------------


Right, the plan above I wrote last night, and here I am in the morning after some success.  But not before an absolute disaster.

I started out with my first mix, and thought I was happy with it... I'd PVA'd the target area, I'd cleared up around the place... The mix was ready to pour and prepare, so I started... whisking with the drill... and yeah it was peaking, it was like a thick double cream... Oh hell... It was just too sloppy... Just a touch too sloppy... but I didn't realise this really until more of it had slopped onto the floor than got ont he wall, and then what had got on the wall was so finicky.

So I tore it and scraped it back off the wall... and despaired... I cleaned all my tools and then smoothed the little I had left around the window over... The inside of the window is now near perfect...

But I was determined to get something done... So I mixed again... this time when it was peaking I added another trowel full of powder... it looked way too thick... But... i got it on the wall... and I got it smooth...

This is the purple area, now its awkward the corner is shitty - because I'm so bad - and there's a lump in the wall just above the window, so I did from the lump down to the floor, around the window sill left... And I got it perfect, this little bit and this mix was by far my best so far...

With the ignimony of the first mix poured into a bag and disposed of (well its in the car ready to go to the tip) I was really happy to get at least this small slice of wall done... and so smooth, its really nice, I got it on, got it smooth, let it cure until tacky then I stroked water over it and polished it up, its really really nice.

Tonight therefore I have the same process to follow and its going to be in 3 steps, baby steps....

So, the first thing is a thicker mix above the bulge to fill the wall in.

Then another mix to go below the window sill smoothing it all out.

Whilst the second mix is curing, I can polish the first, and then whilst the second is curing I just wait.

Then polish the second and make sure I'm happy with all this.

Finally for tomorrow, its a bigger mix and I do the wall area number 3... it'll probably go right to the radiator as well.

This will be a clean up and then polish, and I really really need a thorough clean, there's plaster everywhere.

I need to scrub the skirtings and the windows.  This third area being done marks the end of my plastering before Xmas, because I've found a dodgy area of the walls which I need to check with the father-in-law about - I think the lathe behind it is rotten.

And I can get the desk up, now there are two very good reasons for putting the desk up, the first - the one I'm most excited about getting a look at - WarThunder patch 1.37 is out... with whole new progres model.

And then DayZ stand alone has his Steam - finally - as an open alpha for £20...

WarThunder V1.37 - Impressed

Had my first flights in WarThunder 1.37 last night... I like the new progression system... Also I noted there's a large increase in the numberof crew skill points one can receive.

I flew every nation, and had wins for them all, plus a loss as Germans and a loss as Americans... So I flew out seven times.

I'm a little preturbed I already own the 109 F3 and F3 Trop...but have to research the new 109 F1's... lol

WarThunder - Ranks to Tiers in Next Patch

Out with the ranks and in with the tiers... This seems to me to be going under the radar, or maybe its the elephant in the room....

But Gaijin are changing the progression system in WarThunder, from being Ranks, where by I'm like Rank 12 over all, but Rank 9 in the British... Rank 5 in the Americans... etc.... and the Rank in the individual airforces gives me access to certain planes...

Now the system will be to research fully the previous plane and then go up a plane, and the plans are divided by tier, so there maybe two planes to get through at tier 1 to get into tier 2.

Of course I've seen how quickly you can outstrip your crew skills to fly too high a rank aircraft and get your ass handed to you....

But, this new system is not being mentioned anywhere...

We can see it already in the test server.. and others have covered that better then I...

But even on the live server we can see the start of this system in the unlocks of equipment for each plane, you can see on the left as you unlock the rows, each row is a tier of gear.... Well now each tech tree row will be split by tier.

Now the top current rank is 20.  But it appears the top tier is V... that's a lot of compression put onto the planes...

However, over all I'm going to accept this with welcome, because I think I'm one of those players who has out expanded crew skill and equipment unlocks with my rank... making my game play poor.

This new system will make me rethink, and revisit, things.

Minecraft Texture Pack Selection...

I've been looking at texture & resource packs for Minecraft, Dataless882 (check him out on YouTube) led me to my eventual selection "Misa Realistic".

WarThunder - Patch 1.35 - Economy Fixed!

The latest patch of War Thunder... Hmmm... Yes... Yes....

THEY FIXED IT

*cough* sorry, I got a little excited there, yes after walking away at 1.28/1.29 with the terrible changes they made to the economic model, I've been back with 1.35 and I've made a profit.  I've even lead the score board a couple of times, and I've had fun doing it.

I've bought the new BF109E-1, which complements my German Fighter line where I'd rather have been using the Do-17's than try to use the BF110, or the Italian shite.

I've also upgraded a couple of my British crates with all their upgrades, as I can afford to, and from 250,000 lions at the start of the weekend, I spend down to around 95,000 and I've played only a handful of games and got back up to 200,000.  I consider this a miraculous turn around.

And its made the game fun again.

The visuals are just as good, though on the latest build I am seeing some horizontal tearing as I'm scanning the camera around, but then this is the first time I've run WarThunder on this rig with Windows 7, it used to run flawlessly on Vista on the same hardware, but I've not even bothered mucking about with the settings (yet).

The other new plane I bought is the Spitfire IIa, new in line, and I took it up for a spin on the Battle of Britain Event Mission, now I posted my Spit Mk I success as a youtube video some time ago, however, things did not go as swimmingly in the brand new - low crew skilled - Spit IIa.  No, I climbed to altitude in the usual way and then got bounced by four 109's of various models, the final blow going to a BF109E-1, but I managed to take one with me even with a shredded wing.  Dowding would have been proud - well, perhaps not, but I tried.

World of Tanks - Grinding to the T34

I've reinstalled the machine with Windows 7, and sure enough all the networking problems I was having have gone away, but I'm pleasantly surprised with how spritley the machine now is.

Vista must have been really really in efficient, for example, running World of Tanks, I would run her only in medium to low settings to get 20 fps...

With no changes of hardware, just an update to Windows 7, and I'm now in 1680x1440 resolution, high settings for everything and I get 50-55fps.

Its so pretty....

That was the main victory of the evening, which I spent enjoying with lots of rounds of World of Tanks.

My first discovery was something I had spotted before, but I think will take the next patch to explain, you see you now get missions in the game.  Or at least the rewards for the mission, but you don't have any way of knowing what the missions you have are.  They maybe listed somewhere, but no-where I've spotted in game.

In patch 8.8 however, which is coming soon, WarGaming are adding a way to spot what the missions are.

The night however, as enjoyable as it was, did result in a spectacular number of Defeats for me, despite my best efforts.

Sometimes I was simply out played, sometimes I was clearly stuck into odd matches, and I was spending a lot of games in my T-80 Light tank - an odd machine, grinding the XP for my T34.

But my efforts were not helped with derps like this

Or grease balls like this, whom I spotted on numerous occasions sat in strange locations usually at the back of the pack waiting to pick off enemy targets which the rest of the team - whom are lower tier - have softened up for them... Basically reaper creepers, that's the term I'm going to invent for them.

I did get my T34 before ceasing playing for the evening, and I even unlocked the high-pen gun, but I'm not happy with the vehicle, its match making is pretty terrible.

And I'm also stuck with the news that my favourite tank - with my most trained crew - the VK30.01H is about to be turned from a tier 6 medium into a tier 5 heavy?  Go figure, Jingles has made comment about this, but... Argh, I just want to smack WarGaming.

It might be time for a break and return to war thunder, in which I have also been playing recently, and made a level with my Japanese aircraft.

Also still enjoying Elite Frontier...