Tech Tribulations #2: There's a New System in Town

Time for another story from my tech history, this time I point my torch of memory at one of the first larger systems I worked on from start to finish (nearly 10 years).

I think I can say the product name of this particular system, it was called Phoenix and it wasn't a bad idea, to be honest, as with nearly every project I've ever worked on we needed more documentation and design; those to continuously neglected facets of software engineering.  But other than that, the initial design stood up.

Before we dive into that though, we need to understand the reason Phoenix was thrown onto a whiteboard and born.  What was the direct predecessor?

Yes, there was one, this is not a bluesky thinking system, Phoenix was to directly replace a system which had itself been engineered over the prior two years and not really lived up to expectations.

I shan't name that system, but anyone who knows me and knows Phoenix may know the code name it was given.  It was a hybrid of two languages a C wrapper around the physical hardware bound with JNI to a java core, which itself used the ODBC driver to talk to a Marathon database backend.

The idea being that the transactional nature of the database allowed for the representation of perfect state, and in any error it was simply a case of rolling the transaction off and so we had RORI behaviour in a very complex system.

The problem?

It was SSSSLLLLOOOOWWWWWW.

The host machine was (and I think I'm right here) a Pentium II class machine (it may have been a Pentium III later) with 16MB of RAM (and I believe it went up to 64MB by the end).  Am I talking about early 2005 here, we were working with legacy machines in the field which were being refitted from a system itself all written in C but not well understood.

This new modular java, database, transactional system was envisioned to be the future!

The problem?  Still it was SLLOOOOOOWWWWWW......

I've said that twice now, so for effect lets skip forward to about 2006, two years in and this database java monstrosity is not doing well, it doesn't perform well, the team has literally stopped even trying to work in agile scrum manner; the lead developer didn't take well to critique amongst other issues (like it's slow, have I said that three times now?)

Happily however there were two separate development machines, the one on the old old system and a whole other on the java junk.  So it was in the C group I was called into a meeting with my manager, the manager of the whole department.

He had a punch bag doll in front of a whiteboard and asked me to close the door.  Furtively expecting my P45 (to be fired) he asked me to take a look at the board... he had asked two other engineers to take a look through the day, so he was canvassing for opinion.

In the dry wipe ink was a block diagram, eight boxes, a hardware module to talk to hardware, a module to account, a module to host the menu, a module to talk to the content program, and so on, one for logging, one for sending reports back to HQ... These simple systems.

Each would talk to one another with a message passing mechanism, which was a line from each of these eight boxes.  Little did I know then that this message passing system would be the biggest part of my debugging life for the next ten years; that however is another story.

Right there and then this new system seemed like a much better proposition.

The manager liked the feedback from the various folks he had to talk to, and it was decided upon.  He would work on the platform, starting up the system, securing it with a manifest and preventing unauthorized execution of random code and content.

I would be working on the main attract module, this is the main menu from which content is categorized, and launched.

The content would talk to the Game Interface module, which was owned by a guy called Darren.

The accounting module was handled by the previous lead on the java system, and later a guy called Conrad.  Conrad at first though would be doing some other systems.

And there was a smattering of other work, like a logger module, and a build stack and al-sorts.

Work began, but not before the lead on the then current system had to be told her baby was being canned.  She and her main ally were duly summoned, shown the block diagram and where they would be working.  She did not take it well.

As she left the managers office, he was heard to exclaim and his stapler threw across the space and nailed a window barrier, sticking into it with a vibrating twang.

She loved her system, she did not like Phoenix, not least as it's name was a direct reference to something rising from the ashes, her system being declared to have self-combusted in failure.

Amber Valley Planning aren't very good

I think I've come to the conclusion Amber Valley Planning Department are either incompetents' or simply idiots.  Having had to interact with them myself and now in relation to a nearby development, they've demonstrated incompetance which can't easily be explained... Let me explain.

You see a development was proposed, ironically the owners only sought planning permission after the fact, fine whatever get on with it...

So the ground-works and building are complete?... Yes.

The official visited the site?... Yes, there was a woman with a clipboard.

Does the block plan represent the reality on the ground?... No, not even close.

Any school child with a ruler and pencil would be able to very quickly look at two reference points on the ground, extrapolate two lines then compare them to the building work which took place.  They don't match, not even close.

Further more the building work significantly changed the levels of the area in question.  This is not mentioned on the planning permission request at all.

And the newly raised area is several hundred tonnes being held back by concrete posts... No no structural engineering posts, we're talking 4 inch garden posts, with postcrete and concrete weather board.

In the July & August ran we've had they're already bowing ominously.

And I find these two points extremely curious, for you see we were the ire of this department having skimmed 2 inches of soil and spread road stone we had to account for a "Change of level"... yet someone else can dump several hundred weight and build up a new embankment to around 6 feel of altitude?

We presented drawing plans from overhead shots, real shots from Google Earth, showing the true scale and location; it being a photograph after all.  We had to go back and be exacting, that was the order "be exacting" in the block out diagram shown.... Yet someone else can be approximately six feet off level of reality in two axis?

When this goes over, and it is likely to, could it be a danger to life?  Yep.  Will the planning department comment?  Will they heck as like, they just tick their box and waltz off, it is almost criminal in this case.

Either the planning are incompetents', and I strongly suggest they explain their decision; oh but wait, one can not challenge them, only the applicant can appeal?!?

I must therefore air on the side of caution and suggest they're merely idiots, bureaucratic machination maniacs of the paid civil servant ilk just there to tick the boxes and dot the lowercase J's, since the service... Clearly the service is broken.

Want to know the major comment about the works carried out?  That they must ensure the hedge remain for at least 5 years.  This is a hedge in which (at least further up) I found bottles tangled in the roots with a date of 1914 on them, so this is already very much an older hedge... so a) 5 years makes no sense, b) you ignored they changed the level, c) the works do not match the plans provided - and I told you of this! and finally d) they're already subsiding, dangerously so!

Incompetence demonstrated, well done Amber Valley Planning Department, well done.

I am extremely glad my hard earned council tax money no-longer goes to them, indeed I've spoken to my local duty planner, they were highly available very interested in my requirements and helped me out immensely; Amber Valley, not so much.

My Worst Development Argument Ever

I have had a really excellently interesting sprint with the work I'm currently doing, like you know one of those sprints where you get a real technical tooth into the pie of problems.

That however is neither an interesting story, nor one I can actually tell.

Instead I will fling my mind back a score of years and we'll discuss THE WORST DEVELOPMENT ARGUMENT I HAVE EVER HAD!

The problem started with the platform I was then working on, it was a quite low-powered single core Celeron PC base, running a favour of Windows for Embedded systems (I think it was Windows 2000 Embedded), but it was basically used as a host for a C# application stack which itself was more or less a wrapper around a C hardware driver talking to the various light, button and money handling mechanisms in the machine.

This system of ours then simply would process launch, at the shell level, another child process which was the actual game content.  And there were lots of different games we could fire up.

Our menu and the hardware polling all would back off, it was actually only polling for button and light IO and every second polling for cash changes from money physically being entered; so that was slow to update, but otherwise it was fine and backed off.

Being a Win32 environment it was fairly easy to back off most all the threading and just launch the child process.

Our testing showed that we had a 92/8% CPU split, the system would take on average 8% CPU whilst in this active child mode; it had a few spikes and it had a few troughs, that included windows itself.  Otherwise it was pretty much that most all the CPU was available to the child process.

It was therefore with some scepticism that we had the development manager in the content area wonder over complaining that one of his guys was having issues getting smooth frame rates from the platform.  We accepted the install and ran the child process, indeed we immediately saw the issue.

A popular UK TV show game, with a target circling the board...

The target would stagger and slew around, this was immediately explained to us that the system was so busy the game was being forced to drop frames, the frame rate icon the game had programmed into it went up and down like a pair of kangaroo's in the mating season.  This was the whole argument, our game drops frames your system takes too much time on the CPU.

It has to be pointed out that our manual actually said we only make 75% of the system time available to the game, our demonstrating we were making over 90% available was well within tolerance, and this is the only game showing this kind of issue.  We were therefore skeptical about the claims being made; especially about the high quality and tested nature of their content executable.

After an amount of head scratching we decided to measure how often the "Present" function was being called in DirectX, a little hackery later and we had a measure... Was Present being called consistently and then the system not presenting, or was the game itself changing the length of time a frame took, staggering when it presented?

Yes, the game itself was almost immediately measured to be staggering how often it called Present.  So the question came up "When you move these items are you interpolating between the position and so smoothing where the target is?  Or are you moving them a fixed distance each frame?"

"The frames are a fixed length"... The developer and their manager said.  We had just measured and shown that the frames were changing length, we couldn't look at their code, so we'd had to hack about but we'd shown their code was calling the present with the same staggering it was not a fixed frame rate as they said it should be.

We video recorded (literally with a camera on a tripod) measured the frames on the camera and calculated the stutter.  Then measured with our test harness when present was being called and saw a direct match, when they called present the present happened.  Our conclusion, the only conclusion really, was the content executable needed to be calling present more consistently; or their DirectX present be set up to sync with the screen or something.

The argument then began, they insisted that their game was presenting at a fixed interval, it was fixed to 30 FPS; they refused to turn on VSync, their platform did not allow this setting to be part of the DirectX initialise... It really should have been.  But we had explained the staggering, the delay in present being called, neither were o do with the system all appeared to be in their code.

"But the system is taking 100% CPU"

Yes at this point three member of my department all pulled this face:

It has to be said the manager in charge of this developer is perhaps the worst development manager I've ever met, a man so inept he alienated and lost developers at a prolific rate, who spent £40,000 on migrating to Perforce rather than paying his team and just using git.  Who seemed to think he could pick up modern development techniques by osmosis.

The developer himself?  Well, lets see how he handled all this evidence from the camera footage....

That's right, he lost his shit.

Despite myself, a coworker and our manager all confirming our observations that the hitching and stuttering were coming from the child process itself seeming to go idle and so it calling present at differing intervals he clearly took this to be a personal attack.  He stood in our development lab and basically tore into our system; our whole team took a verbal berating at this juniors gob and his manager relished it (this was one of the earliest examples I saw of this manager relishing in his minions going to town on others instead of his reeling them in).

Rocked by their insistence and being in the unhappy situation of having to prove our innocence our manager asked for us to be able to review their code.  They refused.  There were some issues, because our company had just conjoined with the place this developer worked with, they still saw us as interlopers.

My colleague was known to be somewhat more volatile than myself, so our manager left me on this horrible issue, trying to figure out what was wrong.

Come the Friday, and a little exacerbated after two days, I started to decompile their executable.  It appeared to me that their main loop was miss-behaving, it seemed to be just calling Sleep with a fixed value on top of the work it had carried out each loop.  And sleep takes a variable amount of time.... It was not taking a check of how long their update took and then sleeping for the difference to meet the frame rate target and then it dawned on me.... Did you spot it too?

They were calling SLEEP.

The Win32 API documentation literally says sleep is not a fixed interval, it relinquishes your thread for the remainder of your time slice up to or more than the sleep time given, returning when the windows scheduler next wishes to make your thread active.

It still says similar today, the time slept is not guaranteed.

This client process main function seemed to be calling sleep with a fixed value each pass, it was hardcoded into the executable.  If you sleep for X+Unknown, you are going to see exactly this staggering.  Our demo child application has a busy wait, it never slept, it yielded by passing a sleep of zero (which make it give up its time slice but remain ready, and it would more or less always be rescheduled before the fixed frame rate next time point, plus we interpolated our animation example).

This child process was just proving more and more to not be to specification.

Their manager insisted it was to spec... Which was very frustrating, as he grandly declared "I have read the code, I know it is correct" and "when we run it, we do not see this issue".

It felt like a fundamental miss-understanding, none of the folks on the team I was within were being treated with any respect, nor acknowledging our collective experience and understanding.

I stewed on this all Friday, come the Monday things were getting fractious.  This game had to be out!  It came up in the master development round up that this game was held up by our system.  This started the direct antagonism.

Everything I did, everything other members of the team did, all showed this content application was just staggering and stuttering on it's own volition, by design, by intent it staggered.  It was not our system!

To prove this I therefore set about writing a harness, which would just give the client process the same DLL to load, but they were all stubbed out calls, and it would run WITHOUT our system.  The content could be tested locally and checked.  We got one of the same Celeron PC's, just a flat install of the OS and double clicked my harness.... Sure enough the game staggered about in the exact same way!

I presented a A4 page example and showed how the loop in our example application worked, explaining that Sleep is not a fixed time interval and a busy loop should be used with a yield not a time span anyway.  I recorded this with my harness and their game, I presented both recordings too.

The developer went apoplectic....

He literally shouted at me that Sleep was guaranteed to come back after that amount of time, and when he ran my DLL shim locally it ran really smoothly.

He was right, it did, but he had a much different dual core machine with 4GB of RAM and a graphics card.  Our platform was a single core Celeron 1GB of RAM and built-in Intel graphics shared vram, in short very different.

At this point the director who sat between my manager and this development manager ordered that I be allowed to look at their code.

The worse development argument I have ever had then hit a peak, as I walked over, flanked by my manager and their manager.  I lent over his shoulder and pointed to the sleep function and said that is not going to be a fixed period.

That was all I said, he never let me explain any further, he just went MENTAL!  He started shouting, screaming, and called me a few choice names.  He would not accept all the evidence that his loop was not a fixed length, that it was changing from frame to frame, he just could not figure out that:

{

    UpdateStuff();

    Render();

    Sleep(33);

}

Was not going to always take a fixed amount of time, first of all I pointed out that doing anything and then sleeping like this will be the time the work takes plus at least 33, and then there was no guarantee that the sleep would immediately come back.  The windows scheduler would decide when you can come back after at minimum that amount of time rounded to the nearest platform tick.

His manager immediately backed him up and agreed with him, they both talked down to me.  They insisted loudly and angrily that sleep was fixed and the functions they have took such a trivial amount of time they were not worth measuring.... 

Even my manager backed me up here, of course doing a function call, any function, will take some amount of time and they need to take that into account.

I had just had these two idiots literally shouting at me, whilst I had to stay so calm, it took an icy handful of minutes for them to accept the argument that 33+N is > 33 where N is none zero.  It was just fundamental and they were not having it.

Their code became:

{

    startTime = Now();

    UpdateStuff();

    Render();

    endTime = Now();

    Sleep(33 - (endTime - startTime));

}

Slightly better, but we still saw hitches and stutters, they were far far less frequent now.

This massive drop in frequency I immediately and without changing my argument pointed me to the sleep, as I said the sleep is not a fixed time, it was not going to sleep for X and come immediately back, that's not how Windows worked.

Their argument was that Sleep was fixed, that it was guaranteed to return after X.

They were very loud, very obnoxious and very adamant.

We returned to the developers desk:  "Show me why you think Sleep is fixed".

I expected him to bring up some code, some harness, some proof of his thinking.  Instead he opened Internet Explorer, went to MSDN and showed me the Sleep function documentation.

Sure enough it said "fixed interval".  He was so smug.  So infuriatingly smug.  His manager was ultra smug too.

I reached down, scrolled the mouse and pointed to the screen.... 

He was reading Sleep in the Windows Mobile SDK.  He's right on Windows Mobile sleep is a fixed interval.  However, we're not on Windows Mobile are we.

My manager looked at the screen, I looked at the screen, they looked at the screen.  And immediately the developer called me a horrible name, yup, just straight up called me a name.

I have to admit I didn't react well, fisty-cuffs didn't happen, though the way he erupted out of his seat raging I expected the guy to swing for me.

He could not take it, his manager still argued he was right, so invested in their mistake were they that they could not admit their miss-understanding.  The manager always claiming he only hired the best minds, this guy being quite arrogant and the whole lot of them generally being very dismissive of both myself and the department to which I belonged.

I walked away with my head up high.  My manager stood and pair programmed with both their manager and their developer for maybe twenty minutes and a new version of the content executable quietly appeared without any stutter; even when we ran obrut!

It was a horrible moment in my time with that employer, I remember how the guy never apologised, that development manager never apologised and the game went out without any further delay, but they never received any censure for the episode.

Our department also never shook this kind of effect either, for some reason because their manager had gone to bat for them from the off every following time a performance issue arose we had to prove everything to the Nth degree without ever seeing the other side doing the same.  Very rarely was it ever truly our issue.

I have never forgotten, I have never forgiven.

When you foul up, just admit it, owning it and learning from it is far more wholesome than being uptight and obtuse.

No Rack, but definitely an IT mistake

I sat down to write this on Christmas Day, but things went wrong... so here it is a mere month late!

It has been twenty years, so I feel comfortable sharing this one with you.

Christmas Eve afternoon my landline began to ring and ring and ring, foolishly, rashly I answered to hear an unfamiliar accent telling me to explain to them the complete inner workings of an intranet web service.  No hello, no explanation who they were, just a desperate voice asking, nay demanding, this information....

It took me a few minutes to collect my thoughts and realise what they were talking about.  At the time I was contracting in short sharp sprints between projects for various folks, I'd actually written three different web based systems in the prior three months; so i was trying to figure out which of the recent ones it was..... Only to then hear the name of the company and realised this voice was asking about a project I had been contracted to write nine months prior.

It was a very long ways away in my memory.

Not only was my memory foggy but I had ensured everything asked for in the contract was documented,  all the work was carried out at their offices, on their own machine, I didn't even hold a copy of their system.

Above all it was Christmas Eve!!

I explained that not only was I not contracted to support the project (a sore spot, as I usually made as much money supporting a project as doing it at that time); but this project had actually been a bit of a squeeze, for it had been two weeks work and absolutely no more.

Long story short I'd been contracted by one chap, the manager, because his in house analyst was overwhelmed and unable to cope, so he'd had me to quickly deliver this project for them.  It was a simple view over data on a few pages, driven from a server that the analyst knew and who deployed it out for himself.

The manager and this analyst signed it all off as I went, and I documented the whole thing, critically including how to edit the pages... So they only needed to know a bit of HTML, JavaScript and CSS to change any part of it as they felt suited themselves... and I left.... Getting my money a month later and never hearing anything from them.

On a shoe string I thought I had delivered them a real bargain!  Not hearing anything for all that time had made me confident they were happy.

So what was this desperate emergency?

The colour!  The guy wanted to know how to change the colours in the cascading style sheet...

It is Christmas Eve!  And this £100k per year analyst is calling me to ask for a crash course in CSS!!!

To say I was not amused is perhaps the understatement of the decade!

I politely explained that I was available (at an extortionate price her hour) from December 28th onwards and then regular rate January 3rd onwards for contracting, but right now I was not available.  And all this was available through my then solo contractor company.

The reply?  To be honest he was quite rude, at which point the telephone was returned to its cradle and the TV remote rediscovered.

The trill ring carried on until I unplugged the phone... and my mobile started to ring, my inbox fill, and my patience ran wafer thin.

But I was simply not going to be talking to this guy, my opinion all these years later was that the guy was a paper suit, he sat there getting others to do all this work for him, not even delivering to his own manager and said manager was just accepting of this for some internal political reasons I could guess at, but which were lost on me at the time.

I decided however to be proactive, I wrote up a couple of paragraphs on my headed PDF, attached the original contract, attached a proposal for employing me from the 28th for 2x8 hour slots and then from the first week back in January for 5x8 hours as a starting point.  I put this all together, with an estimate and I sent it in triplicate.... To this analyst, to the manager and to the HR department at said company through whom I had worked for them.

I then had the most interesting email, which only reaffirmed my paper suit opinion of the guy.... For this email was from the manager saying "Why are you sending me this on Christmas Eve?"

And I replied "because your analyst <insert name here> is contacting me, phone off the hook on Christmas Eve to ask me about that work I did in March".

I heard nothing more.... Until the week following the week back, I was at another contract and my mobile rang... It was a very polite voice, the manager, he wanted me to come work for them.... because Mr Analyst had left his role.

I asked a few poignant questions; essentially the chap was earning an absolute fortune (for 2002) at nearly £136,000 a year and he did nearly nothing, nothing.... he'd sat on projects, say on code, couldn't script, couldn't program, couldn't even communicate properly.  Quite how he'd got into the role I will never know.

I however simply kindly declined, I was due to go to Egypt on Holiday so I did that rather then try and fix that guys mess.

Story Time - Fantastic Rack Mount Mistakes - Part 33 and 1/3

It has been a very long time since my last story time, and so this one comes to you with an unknown number, I am however well beyond any possible NDA or secrets for this one, so I can finally share (it's been 20 years yesterday).

I had a job in a data center, nothing too special, but I loved it.  The loud room, the air conditioning, the rack upon rack of machine, I absolutely loved it.  We're talking 2002 here, these were the 2U high style of machine I use a lot today, the odd 4U or 8U JBOD box too, loved them all.

I had all my wiring terminated and was beautifully dressed, I wasn't in charge of the whole place, just one isle, but I took my time whenever there was an issue to run new cable bundles beautifully dressed along the cabinet runners into each cabinet top, seven cabinets to a row, two face to face on the cold row and then the back of these into the hot, I had a hot row run by a lovely guy called Jerod next to me, he was French I think, lovely guy (shout out to Jerod if you're reading!).

The other side I had the work bench, a coveted space, between the hot and cold isles were thick plastic strip drapes, like you'd get in a meat packing plant.

We would patrol our isle, identifying any issues or disks showing physically bad, schedule them in for late night fix ups and then to into the communal office where we each hot desked.

Each of us had a laptop and would sit and connect into the monitoring suite remotely, with a big screen we connected via VGA to our laptops and dozens of spare keyboards we could quite often spend the night shifts doing nothing but playing games; Medieval Total War was the order of the day for me at the time; some of you know of my origins in the "game industry" take me back to hacking about and modifying MTW (Sorry Tom, whom I now work with, for fishing about breaking your baby).

Anyway, about three months into this job another tech on the day shift came to me about Isle 4, the far end of the room, he was getting drop outs at intervals during the day, but none at night, he was trying to figure out what might be causing it.

We took a look at his logs and sure enough on his disks he had 250-700ms delays for seek times on his platters, he also had a higher than average fault rate on disks, apparently he was also rebuilding raid arrays a lot.

He asked us in the night team to watch over the kit live; which we duely did.... Quiet as a mouse, nothing not an issue ever in about three months of monitoring.

Meanwhile in day light he was tearing his hair out; I've got to admit to genuinely not remembering his name, but he had long greasy black hair and was from Northampton which he told me more than his name... "Hello I'm <forgetable>, I live in Northampton".

But forgettable as he was I can't forget the day he handed that isle over to me.

Stumped as to what was going on, and frankly pulling rank, he shuffled Jerod down to Isle 3, himself into Isle 1 (my isle) and all new machines were coming into isle 2, which mr senior with disk failures was going to be setting up.

This really irked me, my isle was by far the best presented, when sales wanted to show new accounts around they showed them my isle and they all laughed over the name of the cabinets; we couldn't name the machines, most of them belongs to customers, but we could sticker and name the cabinets themselves, so they all got Bond Theme names.... Jaws, Octopussy, Moneypenny, Q and M all had simple servers, processors more or less and my two JBOD cabinets were Sean and Roger.  Stencilled on in lovely white enamel paint.

So I was hoofed off to isle 4, dark, at the back, no work bench... and immediately set about making it my own, I redressed the cable bundles (which annoyed Mr Senior, though he loved my already done ones, my dressing his was seen as an overt declaration of war).  It's so simple too, cable tie around five, cable tie through the middle to break them into three and two sets then pull.... Today you can get cable combs and I find them so sexy.

Anyway, dressed to impress what was not impressing was the fault rate, and instead of it being the accepted norm; as it had been for Mr Senior for months; it was now a big deal because I was in charge of the isle.

I set about trying to work it out, I could not figure it out.

At night, when I was there it was all fine, the day guys saw nothing; they pretty much just tended the machines remotely, no maintenance happened between 7am and 7pm so it was up to the night guys like me to swap disks, rebuild arrays, fix cables and install new machines.

So after two weeks with the log I got showing all these faults, like what was causing a disk which had worked all night and most of the day suddenly to show 500+ milliseconds seek!  It made no sense.

Anyway, 2002 was coming to a close, December 21st rolled around and I offered to cover a couple of the guys shifts, they had kids it was nearly Christmas and we worked Christmas Eve and half Christmas Day, one guy on on guy off for 6 hours at a time those two days, I took two shifts back to back.

Alone in the office, I watched the monitors and played games.

No spikes.... What gives, no spikes on the DAY I am actually here.

Hold on, I'm here, but no-one else is.

Could it be human interference?  I checked electrical circuits for noise, I checked lights, I checked if the air conditioning was affected outside and in.  I checked everything, no signs, no peaks, no noise.

HANG ON.  NO NOISE!

Spinning disks can be affected by loud noises, specifically by vibrations, I'd seen this demonstrated when doing my Compaq certification training.

There was no noise, could it just be vibration?

We were on the first floor (for anyone in the US, this is the floor above the one on the ground) so we are one storey up.  There was the main reception below center of the room, to the right where isle 1 is would be a hall way void with offices leading off it, under isle 4 would be a toilet and a changing room.

I went down, put the shower on, came back up.... Nope.

Then I took my laptop with me, no wifi, but I could plug it into various office ethernet ports around the place as no-one was in.

There I am Christmas Day, banging doors, flicking lights, flushing loos and then pouncing on my laptop to see if it affected any of the disk activity I was artificially running up.

Nothing.

Defeated I logged my time, handed over to Jerod and went to have my Christmas dinner.  I had 12 hours to think of something for Boxing day.

I walked into Mr Senior asking why there was CCTV of me "Dashing about with my laptop in random offices".

I just said I was trying to check light circuits for his disk issues; he made it abundantly clear they were my disk issues and left me to it.

Boxing day, the sales had started, the building was at the corner of a large commercial estate, there being a newly built Ikea across the road as the crowds rolled in and their stock levels fell they would be due a delivery soon.

I still poked around the office and the eureka moment came on the 27th December 2002.  For a large lorry was rumbling past the office, I had to wait for it to pass, a massive blue IKEA clad lorry.  I went into the office, logged into the monitoring and sure enough there was a trace of a large disk issue.  Times 4 minutes before; when that lorry went past!

I didn't hear it, I didn't feel it... But had the disks?

I waited and watched when a few lorries passed during the fairly quiet week between Christmas and New Years, nearly every heavy lorry going back resulted in some affect on the disks, vibration was being carried into the ground and I guess up through the building.  I will be honest, I could not tell.  But every spike I saw was timed with a lorry.  In fact I soon let Jerod in on my idea and plan to fix it and so he watched and I monitored and when he came in I could tell him when a lorry has passed!

I was not about to shout about this to Mr Senior, instead I set about fixing it.

I ordered a mat of 1 inch thick rubber, the stuff you mount washing machines on in your kitchen.  I already knew the racks pretty well, they were bolted at each foot, I'd need a torque wrench to unbolt them and I could use two of the hydraulic scissor platform lift trolleys we used to move machines about to lift the rack ever so slightly.

I didn't want to do this alone, so Jerod was roped in with the promise of a take away pizza from the glorious; but long gone; parlour we loved.

Mr Senior handed over to me and Jerod that night, it was not uncommon for two of us to be on at night, especially if there was work to do fitting something out.  It was normal to have two people when we were lifting machines too.  But Mr Senior would have been apoplectic if he'd know what us two kids were about to do.

I unscrewed the first rack from the floor, swept the dust out and used a wooden baton to bridge the lip to the metal of the scissor lift and I cranked it.... The cabinet moved, I nearly wet myself as it looked like it was going to topple, then Jerod jacked his side up on the other side and it came level, with an inch to spare I slipped in two of the pads.   He slipped in his two and at a shout over the loud AC we lowered.

I then used a phillips head screw driver to puncture the rubber and a knife to dig a bit out through each bolt hold and fastened the bolt back through, not too tight, just tight enough.

It was sweaty work in the hot isle as I was, but we did the first three of seven that night.

I went home and decided to pre cut the squares and use a drill to cut the middle out of them,the next night all seven were done.

Our disk fault rate when to zero.

That was my last time as an IT minion; I went back to programming soon after, and Mr Senior never was told what we did... His training should have told him.

What did Don do?

First things first, we're going back to the latter half of the 90's here; and I've mentioned Don before in one of my older posts, skidding into hero mode.

Don was a programmer, he was kept in a little plexiglass bubble of his own and spent all day in what looked like vi (certainly a terminal based text environment) writing what I think was C for the NCR minicomputer running Sco Unix I mentioned before.

Whether this is actually true, I don't really know, for he spent a lot of time sitting with his hands crossed over his head waiting; at the time I didn't know what he was waiting for, at home I had a Pentium 100Mhz rocking 16mb of RAM and running Borland Turbo C++ for DOS I had a lovely little progress window showing me it was compiling; I had not met the glory of GNU yet.

So Don would stare at a massive (like 19 inch) monitor, he had a Wyse VT100 next to him too, and he would not stare at that very often.  He would also make copious notes on paper and seemed to program very deliberately, looking and reading through the information at hand.

He did this, almost without interacting with anyone else for the whole year I worked with him; looking back on it I wished I had been more proactive and just asked him more about it, as he was clearly programming C around a Unix system, which is a) uber cool and b) uber cool.

It always struck me quite how long he thought about and maybe designed anything before acting.  You could say this was because he was mulling and considering the problem, he was being a good engineer, with forethought and wanted to get things right?

No, see I think he was really just in fear of how long the compile would take...

Just like me with the project I'm working on twenty five years later.

Final Thoughts for March - Apple M1

I've been looking at the new like of Apple equipment, now I'm not an Apple guy... I only recently acquiesced to buying a second hand Mac Mini (Core i5, very old) not really to play with the old version of Mac OS it could run, but to use it as a Linux box.... Of course I took a look at the Mac OS and the hardware.

It is very neat tidy design, but awfully bespoke, terribly difficult to repair, but not impossible, I actually was able to fit new SSD's, upgrade the memory and fix a failed fan.

That's the bare minimum I'd ask of a piece of kit, to be able to change the storage, upgrade the RAM... fix a fan... Hard drives, be they mechanical or solid state fail over time, they literally wear out.

Hearing that some Apple models have their only storage literally soldered onto the main board was a real turn off, how can such a thing be upgraded or repaired?  Well it can't.

And this ability to repair something is pretty key to me, and I presume a lot of other people.

Soldered on, fixed in position, storage is just a no-go... And as I understand it this is how the new M1 Mac's all work, they have a fixed amount of storage, no additional storage interfaces (SATA etc) sure you could use an external USB drive, but that's... that's not going to stop the main boot drive taking a battering.

Worse still I've anecdotally heard about the massive swap usage on the M1 Mac's, which prematurely ages the storage.

It is the perfect storm to leave a lot of electronics in ruins and the waste of resources and environmental impact of all this just has be aghast.

Stupid Things I've Been Told

Throughout my life I've had moments where people simply nay-say what I've told them... This became a very real trope (until I took up a marshal art and became very good at it) in my life and lead to a loss of confidence on my part, today lets take a look at some of them which I still remember:

English (1991)    AMBULATE

I used the work "ambulate" in a sentence, I was 12 or 13.  My mother immediately said "There's no such word", and I said "there is, it means to walk, and it's where the word Ambulance comes from"... She flat refused to accept this and really made me unsure; she was wrong, she has been wrong about many things, but this was one of the earliest times I actively recall her being dead wrong.  Her response also made me second guess myself, so much so that I sat and read the dictionary, other kids at school ribbed me about this for years after, but I read the dictionary.  I was right, ambulate is a word, it does mean to walk.

Engineering (1985)    METAL SPARKS

My Dad was in the door way of my Grandfathers garage, angle grinding part of the front of the family car, and there were metal sparks flying all over the place, to the left of the closed wooden door was a fabric cover over some garden furniture.  The sparks were flying and landing on this dry fabric.  And I said "that's going to catch fire".... I was 7 years old and I could see this cascade of burning sparks would set light to the fabric if it carried on.  My Dad turned to me and said "don't be stupid, there's nothing in a spark".  As though it was a zero weight particle an ethereal nothing, of course, we all know that a spark is a red hot but admittedly tiny fragment of metal, not only could it have set light to the fabric, but it could also have gotten into someones eye... But, I was seven, I was clearly wrong... No, no I wasn't.

Electronics (1992)    CD PLAYER

I was sat in the library with a selection of the annoying, ignorant, bully children who were my class mates, when one of the boys Richard ventured something about CD's, about their spinning... of course CD's spin, we all know this, that's why they're round and the lazer is set to slide back and forth.  Richard wasn't known as a braniac, but he was right.  Immediately a boy called Zahid took the utter and total michael out of Richard, really horrible child this Zahid was and he was one of a bunch of morons in my class... Martin, Martin, Paul and Zahid were a group of utter moronic and bullies; each with their own insecurities; and they took it out most days on me, but this day they dragged this poor Richard lad into it, who looked quite abashed, I decided to stick up for him... Well, never had I been so quickly and thoroughly insulted by a moron who knew nothing but how to ride his mouth.  Zahid, I hope your now very bald head knows just how wrong you were, and what a moron you looked flat arguing that CD's don't spin, because they do.

Computing (1984)    TYPING

Sat before the first BBC Micro that the school had bought and the teacher was trying to explain LOGO to us, he was leaning over the computer typing one handed upside down, and I could tell what he was trying to type... For I had had a computer at home for the last year, and I'd been typing in simple BASIC programs, so this LOGO thing looked very rudimentary even to my six or seven year old mind.  So I offered to do the typing for him, he was copying off of a sheet, he'd have been able to tell the class the command and then see me type it in and the turtle robot (or just the screen) complete the command.  And better yet, I'd have been so much quicker making the class feel far more interactive.  This teacher, Mr Allison, turned his eyes down on me like I was a dirty stinking urchin (which I likely was) who not only couldn't understand what he was doing, but it was far far beyond anything I could ever do with a computer; see we were a school intended to produce manual workers, no-one expected a kid from that school to end up being a programmer... I'd hazard a guess that a fair few are now, and I look back on that day thinking... Why did he say I couldn't possibly type?  When I could and can....

Computing (1993)    FLOPPY DISK FORMATS

I was always curious about computers and when my senior school created a "lab" with twenty Acorn A3000 computers I was eager to give these 2megabyte machines a whirl.  My machine at home was an Atari ST and it's floppy format was compatible with the Acorn machine (they both being nearly perfect compatible matches with MS DOS's format on the IBM PC).  But this lab was always closed, not always locked, but closed... But I started to let myself in; and one day I was caught in there, learning, programming and typing notes, by one of the teachers; I don't remember this guys name, he was never my actual teacher, but his lab was his coveted land and I think he was a little put out that I was there with two of his machines going doing different things at the same time, not just that, but I was doing things so much more quickly because I'd learned that the Acorn OS came with a RAM disk, and with 2MB of RAM I could assign 500-720K and basically copy the Floppy disk straight into RAM, work on it incredibly quickly, and then sync the data back to the disk... So I could do the work I was doing incredibly quickly.  He didn't like this, he didn't want to learn, he just wanted to make me look a fool.  So he asked me what the disks I was using were.  So I showed him, "You have an Acorn at home?"  No, I have an ST... Silence, he just stared at me, and he paced off to his desk and pulled a disk out his desk... "Go home and make me 10 copies of this".. and he handed me a box of blank disks... The disk was Lemmings... This teacher was literally giving me homework to commit copyright fraud... I didn't take that disk home, but I did return to his lab... turn on all the machines, and copy the disk once.. then with the two copies copied it on two more, then on the four machines I made the next four copies, and then I had the last two machines running the write of the disk as he walked back in... A mere ten minutes later.  "That should have taken you all night!".... Why... it's binary maths... 1 becomes 2... 2 becomes 4... 4 become 8.. here's your last two... "Aren't you the IT teacher?".... he just stared as I handed him his contraband and left the room.  I didn't like the man, but I was a black-belt in karate and knew who I was by then...

Phantom Recruiter Calls

This post actually got corrupted... and wasn't that funny, so I'm just nuking it.

Basically the guy called me, and I had NO idea who he was.... wasted my time, as did this post.

Great Rack Mount Mistakes #7

Today's story in the annuls of problems in IT comes from a guest editor... Mr B.... And Mr B (no relation to anyone in other stories given monogram names) works as the sysadmin and developer for the whole set of systems with his employer; unfortunately this means "it's all his fault".

So what went wrong?  Well, over night the site had a power cut and though they have a nice server, they don't have a power back up, so that server went off.

The server is essentially a java host, specifically hosting Tomcat, and it reaches out to connect to a set of third party endpoints via a restful API.

You'd think no big deal, start up get running and keep running, except that third party don't force a disconnect upon a new edition of their interface API, if you're connected to version 1.0 then you can and they will happily leave you connected to version 1.0, even if they release interim updates, add new calls and quite what got Mr B today remove a call or two.

Your session ending, and then I presume all sessions of that old version, would free their server provisioning to de-allocate the old version.  But to force users to migrate upwards in the chain their published API (so think the end point here in whatever flavour you wish) declaration changes.  Such that you re-download it upon re-connection and that's your new flavour of the month API.

The problem?  It didn't work.

So Mr B had to set about debugging this on the fly, in a live environment, which was down.  And he went through the three stages of technological grief....

1) Denial:    "This is completely illogical, my code brings down their interface, which is the only thing we connect to, it must be right, they can't miss-match them, so this must be my side or the gods are against me".

2) Investigation:  "Read the logs, make a change, nothing seems to work, the gods are definitely snickering behind that cloud of steam now".

3) Realisation:   "If it's not me, and it's not the system here, it must be their side, the huge multi-billion international must have published their API spec with a mistake or miss-match.... click.... YOU FUCKERS!"

What was the actual problem?  Well, the third party published API was actually wrong, the downloaded specification still contained several calls which were removed, when the services Mr B had written came up they checked each end point and found several calls defined which did not respond and so his software, correctly, reported that the endpoint was offline.  They were, they didn't exist anymore.

His fix was to literally tell his stuff to ignore the multi-billion dollar international service providers API spec and to "download" a copy which he hosted locally, with his own edits to it.

Now, he's a tiny fish in a huge pond here, even if he reports this miss-match said multi-billion dollar international isn't going to hear him, and by the time he does they maybe several months down the line, and other folks may have spotted this problem.  He maybe listened to, but he essentially doubts his voice would be heard.

The problem of course being how to abate this issue in the future?  How to avoid this stress?  For at one point he did say "the company is done for", because literally everything was offline, all their services were down.... And of course everyone will blame the little guy doing all the IT, they won't think that the multi-billion behemoth entity could possibly publish a wonky API spec, most of those shouting at Mr B with mouths frothing wouldn't even know what he meant when he explained this to them...

The fact that he's identified this issue, resolved it, and everything is back up within two hours won't be remembered, the glass will remain half-empty, and so it'll only be remembered that on the 3rd September 2020 Mr B's IT suite went offline.

Drop Ships Dilemma

I'd never heard of "drop shipping", to me it conjure soldiers in heavily armed space-shuttles being flung pell-mell at the surface of an enemy planet with the winds of plasma and fire burning all around them.

However, what it actually means; apparently; is that you've bought something - say a chair - off of a reputable site like Amazon, but you did so not from them but seller on their platform.

This seller then doesn't actually sell you anything, what they do is contact their own network of suppliers and they buy the item for you and get it shipped by that third party to you.  The drop ship being that third party, to whom you don't exist, they deliver to your address as though you and your address is the middleman you met on whatever selling platform you chose.

And apparently, this is legit on some selling platforms, Amazon, Ebay etc etc.  The former put some extra steps in there, such as if you have an issue that middleman has to accept the return and handle the shipping, but this is a nightmare, because you use something like Amazon or Ebay for the protection the service affords, the expectation of easy and care free returns.

The trouble being, this middle man is only there to cream a little commission.

So my issue was a chair, is a chair - this is still going on - which I ordered from Amazon on the 13th, I didn't even clock it wasn't via amazon themselves, this chair has a brand and a name and was set to be delivered on the 20th.

Which came and went no sign, also no tracking, when these drop shippers do their thing they can keep their cards close to their chest, Amazon just ask that they've dispatched when they say they have ditto for ebay.

Trouble was this middle-man tried to tell me that they had called and left me a voice mail, and I'd not gotten back to them... .except, I don't have a voice mail, don't like voice mails... so they lied.

During this discussion they then said (bare in mind this is the 21st, a day late and eight days since ordering, which said "dispatched") it'd not even been dispatched, indeed it wasn't expected into stock with them until the 27th.

Silence.... I want my chair, I want to know when it'll arrive.

I still don't actually know whether this is a drop shipper or a scam.

"We have another chair, I assure you it's better, more expensive which we can get to you tomorrow".

Now, just because something is more expensive does not make it better, and I spent time picking this chair out on Amazon, it has all the bells and whistles I want... four major features and a comfy shape.

This middle man is persisting to send me details of another chair, and "if you agree I'll send that next day delivery"... this seems like a scam to me, but I don't even get to choose because the information never arrived.

I never saw the info, so never okay-ed it, I never wanted anything except the item I ordered.

Just now, I'm in the kitchen and I hear the letter box flap, and a van drive away... there's been no knocking, no noise made, just a note through the door "left with neighbour".

Hate my neighbours, but go find out what this delivery was... turns out to be a chair... Not the chair I ordered.

A quite ugly square one... with two features I ordered missing, made of plastic not steel, and a manufacturing fault along the top cushion.

So I call this guy up, tell him "I was expected to wait for the item I ordered, I never got the info, never asked for this item, it's the wrong chair and missing features and made of the wrong material".

"This is a more expensive chair" he says.

It certainly doesn't look it, and even if it arrived with gold bullion stuffing the cushions I'd return it because it's not the chair I ordered.

This guy is getting on my nerves, and that's when I realize I've been drop shipped, this guy is in an office himself, this is "his business", but he's clearly at work doing something else, he's not a chair saleman, he's just acting as a middleman.

So back to the selling platform, Amazon... who... seem to be quite happy to let 48 hours pass until they will even talk to me, to give the guy time to reply to me...

I therefore am out of pocket, still without a chair, and frustrated by this whole mess.

I think Amazon need to make a very much more clear distinction between the items sold and delivered by them and items being drop shipped like this or worse.  "FROM AMAZON" in a big clear type.

There's even issues there, for instance I've seen resellers listing parts like say "Intel CPU's" and it'll be listed as "by Intel"... but it's not being sold by Intel at all, sure it's made by intel, but "John Doe Computer Tech in Driotwich" are the seller on Amazon, and really we need to be told that more clearly to avoid the kind of situation I'm now in.

Brain Dead Boring Boomber Bastards Next Door

Its been over a month, so I've come to terms with something... Some people are just horrid.

We'll just dial back the clock, 1945.... VE Day... All four of my grandparents were serving in the war effort, Solider, Sailor, Nurse & Land Army Girl respectively.  It was their war, their life, and I respect that immensely.

I made a video to that effect a head of VE Day.

However, with the lockdown, my wife and I, being in at risk groups didn't plan on anything for a street party out front, not least as our neighbours, as already mentioned in these pages, are quite abhorrent people breaking the rules and putting themselves and others at risk, but we also had no idea there were any such plans a foot... After all, why risk it when we and all of them are in at risk categories?!?!....

So, we sat it out, rightfully and with respect, I do spend a lot of time with my mind cast back to the world wars and I don't think any of my neighbours, unless there's some secret military historian in their ranks, can touch my knowledge of the struggle and suffering inflicted.

It was therefore with some surprise that I've heard on the grape vine how "disgusted" our direct neighbours were of our "ignoring VE Day" and "disrespecting the spirit of the day".

These fools haven't dared say this to our face, these brain dead boring boomer bastards won't face me and say a word, and you know what... that makes them pretty horrid in my book.