Saturday, 10 October 2020

Twelve Drives Exit - Only Ten Drives Survive

 I've spent the morning doing a bunch of hard work, like using a surface compactor to lay a 60 meter gravel track... I was knackered come lunch, so I decided to have a play in the server world.

Now, some of you may know we're between properties at the moment, this means I had to take all my servers offline and move them.

However, I've been desperate to get the 32 core machine back online.

Booting up just now though and a couple of the drives had gone bad, like physically bad.  As such my ZFS pool was just a total write off, so I've decided to restore from my offline back up.  There's not actually that much data on that cluster, it was only running raid 2.... And with two disks dying in a simple move, I thought it better to go for raid3.


But remember folks, RAID is not a back up.  My important server is a triple mirror Zfs pool.  I can lose any 2 of the three drives over there, and they're brand new nice WD SSD's... plus the server is ONLY turned on for backups.

This serer is my scratch working/coding project server, on which I host my build slaves and nodes in network tests etc.

Thursday, 1 October 2020

Frank Herbert....

I started to re-read Dune last week, I've read a good piece each night and finished the first book... This is ahead of the new Film adaption coming out later this year.

I first read it in the late 90's whilst at Uni, whilst reading a bunch of other classic's such as all 2001 and it's sequels.

What I had utterly forgotten was just how easy it is to read, it's what I would describe as a smooth read, you feel that you could just keep going each chapter on chapter.

Thursday, 24 September 2020

The Racist Editor

I was just reading about Arati Saha, her being highlighted to me by today being her 80th Birthday and google giving her a doodle.  My target to learn about her was of course wikipedia.

Unfortunately as I was literally watching there's some racist person on there, sadly someone in the UK, who use Sky Broadband editing the page in some disgusting ways.

For example the page clear is of an Indian woman, awarded by India etc etc, she's Indian, absolutely unequivocally.

Yet this little oily turd, hiding behind the supposed anonymity of their IP address was editing it to say she was English, Scottish and Welsh.  Similarly they were defacing the page to say she was alive in 2021 and had the Nobel prize... every edit had a slightly more western bent to it.

What kind of bored, pathetic, moron defaces a page, clearly only highlighted to them by the google doodle, at 9:10am of a morning.  Can't they find anything better to do with their time?... Like post themselves to Bangalore and learn about the culture, exquisite cuisine and fascinating people of India?

Here's the "Whois" on their IPv6 address, I'd ask you all to do as I do, and send an email to sky reporting the abuse of this page by that IP address, see if in this sea of abuse and racism we can get some action from them?



Sunday, 6 September 2020

Forgot to Ride a Bike... Or?

I'm totally confused.... For you see, they say once you've learned to ride a bike you never forget, and I've not forgot, not one bit... So what's the problem?  Well, the problem is that the bikes have changed!

No I'm not talking electric, no I'm not talking mountain or anything like that, I'm talking about the brakes.

You see, I'm right handed, therefore my primary braking hand is my right hand, in a car I brake with my right foot, on a bike I want to brake with my right hand, this is how it is, how it was and how it always should be.

This should be your rear brake, you brake with your rear wheel first, so you don't lock the front up and go over the handle bars.

It's simple safety and ergonomics, and sure as a kind I remember lefties swapping their brake handles.

But... but... we've bought a bike from France... From those duplicitous Francophiles at Decathlon... And the primary brake is the rear brake, but they put it on the left!!!!

They literally put it on backwards.

It does my nut in, that I have to crack the plastic on the molded handle and swap them over... And no, I can't just remove the wire and swap that, because the rear brake wire is a physically different material to the front!

It is infuriating.

Friday, 4 September 2020

Great Rack Mount Mistakes #7

Today's story in the annuls of problems in IT comes from a guest editor... Mr B.... And Mr B (no relation to anyone in other stories given monogram names) works as the sysadmin and developer for the whole set of systems with his employer; unfortunately this means "it's all his fault".

So what went wrong?  Well, over night the site had a power cut and though they have a nice server, they don't have a power back up, so that server went off.

The server is essentially a java host, specifically hosting Tomcat, and it reaches out to connect to a set of third party endpoints via a restful API.

You'd think no big deal, start up get running and keep running, except that third party don't force a disconnect upon a new edition of their interface API, if you're connected to version 1.0 then you can and they will happily leave you connected to version 1.0, even if they release interim updates, add new calls and quite what got Mr B today remove a call or two.

Your session ending, and then I presume all sessions of that old version, would free their server provisioning to de-allocate the old version.  But to force users to migrate upwards in the chain their published API (so think the end point here in whatever flavour you wish) declaration changes.  Such that you re-download it upon re-connection and that's your new flavour of the month API.

The problem?  It didn't work.

So Mr B had to set about debugging this on the fly, in a live environment, which was down.  And he went through the three stages of technological grief....

1) Denial:    "This is completely illogical, my code brings down their interface, which is the only thing we connect to, it must be right, they can't miss-match them, so this must be my side or the gods are against me".

2) Investigation:  "Read the logs, make a change, nothing seems to work, the gods are definitely snickering behind that cloud of steam now".

3) Realisation:   "If it's not me, and it's not the system here, it must be their side, the huge multi-billion international must have published their API spec with a mistake or miss-match.... click.... YOU FUCKERS!"

What was the actual problem?  Well, the third party published API was actually wrong, the downloaded specification still contained several calls which were removed, when the services Mr B had written came up they checked each end point and found several calls defined which did not respond and so his software, correctly, reported that the endpoint was offline.  They were, they didn't exist anymore.

His fix was to literally tell his stuff to ignore the multi-billion dollar international service providers API spec and to "download" a copy which he hosted locally, with his own edits to it.

Now, he's a tiny fish in a huge pond here, even if he reports this miss-match said multi-billion dollar international isn't going to hear him, and by the time he does they maybe several months down the line, and other folks may have spotted this problem.  He maybe listened to, but he essentially doubts his voice would be heard.

The problem of course being how to abate this issue in the future?  How to avoid this stress?  For at one point he did say "the company is done for", because literally everything was offline, all their services were down.... And of course everyone will blame the little guy doing all the IT, they won't think that the multi-billion behemoth entity could possibly publish a wonky API spec, most of those shouting at Mr B with mouths frothing wouldn't even know what he meant when he explained this to them...

The fact that he's identified this issue, resolved it, and everything is back up within two hours won't be remembered, the glass will remain half-empty, and so it'll only be remembered that on the 3rd September 2020 Mr B's IT suite went offline.

Wednesday, 19 August 2020

How much CPU?

This is a story from around six years ago, when I was given a virtual machine host on a server by the IT guy and he was a little perplexed that I actually used it... A LOT.  But we also found something out about the way his session management worked.

The host machine was some Dell server with a pair of Xeon E5 class processors, I don't know what they actually were, for I only saw a very limited guest virtual machine.

My virtual machine was some debian based distro, I think Ubuntu, but running gnome not Wayland.  because we were working on some gnome hosted front end.

Anyway, the machine was a continuous integration machine, and I immediately set to making it build the software every time there was a commit to the repo and also to build it overnight and give us a daily build and smoke test.

The problem?  Well, this was quite a lot of work.  I'd been provisioned to use two cores on the host machine and I basically maxed them out, building the whole system software and then overnight building the whole toolchain and then the system and packaging it after running a series of intense tests basically took 100% CPU for about 3 hours.

And the IT manager was a little befuddled quite what was going on when he kept getting told there was massive activity on his machines overnight.

He wondered about hackers, or some vulnerability which had let them in, of course being a windows guy and the host being windows he just blamed Linux, and I sat pondering the problem and then simply asked "Do you know what the machine is being asked to do?"

And I showed him on my workstation, which was 8 cores and 32GB of RAM... and he blanched at the amount of work... "You're only building software, why does it max out the CPU for that long?"

The lesson?  Don't skimp on your continuous integration!

Friday, 7 August 2020

Ten+ Years of Phoenix : A history of a Project

The Phoenix Story.... To be clear I'm not talking about the book:


No, I'm talking about a project I worked on for just shy of 10 years, which started off as the idea to really sort out the product line we were working on, software that did end up on thousands of machines in its time; was ported to at least five different hardware platforms; but which ultimately needed replacing almost before it hit the market.

Phoenix was conceived, I think, by my then boss... We'll call him "Mr B".  And it stemmed from frustration with the previous released to the public project, which was written in a bit of a shambling mess of bad C and only one chap knew how to work with it when I joined... It might have been a little bit of C++ (I remember the chap reading the first edition of the STL book)... And then a failed attempt to make a new system in Java, a bit of C++ and some SQL.... but mainly java connected to a marathon database.

Marathon you might ask, was an open source variant of DbaseIV and had been used by one of the "senior" software engineers in her prior role.  If you google for marathon database today you'll be hard pressed to find a link to it.  There's a reason for this.... It sucks.

But before we go into the technical's here, "Senior" software engineer.... lets call her "Mrs N", really was a bit of a number, when I joined as only the fifth permanent staff (and ironically I joined as a technical documentor not a programmer) she made out she was very senior and really made out that she knew everything.... She effervesced this air of knowing and knowledge, she very much talked the talk... But did not walk the walk.  It wasn't until many years later that I learned she had only been there two weeks longer than me, a mere ten days and that she wasn't a "senior" anything, she was leveraging herself to the management regularly to make herself sound indispensable... She wasn't.... Not at all.

Back to the technical's, this marathon database was to hold ALL the information about the system state, literally it was a state machine without the words "state" or "machine" being used.  The software created a connection to the marathon driver (I can't remember if it was ODBC or ADO) but the up shot was the system ran incredibly sluggishly.  Booting windows and then the software took upwards of 5 minutes, it was a real bane and a pain to work with.

Everything, and I mean everything, went through a stored procedure; which Mrs N wrote, if she'd not written it, then you were not going to get that data and if the result of the stored procedure was not what you expected then you had to submit a new request, she never really went back and fixed bugs, things were either the way she wanted them or you had to ask for more things... so there was massive bloat and creep in this code base almost from the get go.

Bless him Mr B started to try and do code review, he had an idea of trying scrum, but it wasn't scrum, he didn't make everyone be quiet and talk one at a time, quickly then move on.. No.... And I've touched on that shambles before.

With Mrs N ruling the roost Mr B quietly (and sometimes not so quietly) retreated to his office and he would hire folks.  And at some point he hired a contractor, Mr J....

Mr J in the short term was okay, he did his thing, he knew his stuff and he had good ideas, but he was hard to work with.  But he knew this new thing called C#.  And quietly in the bosses office, on his whiteboard was born "Phoenix".

The moment of Mr J presenting this to the boss he was moved onto this super secret mission to make the new new system in C#.

Which left a development vacuum.. Mr B needed someone to use in order to hide Mr J was not doing shitty java to marathon mouth to mouth.... I was thrown to the lion that was Mrs N.

Technical documents out the window, into that position they needed a programmer and I plugged away at that system helping the web developer do an integrated menu and control system, but we constantly battled Mrs N.

Mr J and Mr B however kicked off this C# and soon they had a state machine, an actual state machine by Mr B and an event passing system by Mr J... the special thing about this event passing system was that is worked cross-process, so you could stand up a C# application connect it to listen for a message... then stand up a completely separate C# application and it could send the message to the first... This allowed for a highly modular design with each part of the system mapped down to a state machine and a connection to the message bus.

It made perfect sense, it worked, it was better than using the slow assed database as the inter-process communication bus and importantly the events could be monitored, so you could introspect what was going on.

This was Phoenix... State Machine, Inter-processing Communication Messages, Highly modular.

I would assume at some point the plan was to make things like new features modular so a customer could buy a base configured machine and opt to allow it to do whatever other functions down the line, like add accepting credit cards or add printing a ticket, whatever.  Modularize it and add it on later, or critically remove it.

The problem?  Making this highly modular effectively meant everything was highly asynchronous, and there was no way to synchronise the ballet of messages going between applications, after a few versions even something as simple as booting the module executable's in the right order became a tiresome mess.

But, the initial versions went out to customers and they did buy them, and so the product lived on.... However, a highly async process stack, running on a single core Celeron processor... isn't particularly asynchronous, there's actually a huge amount of linear ordering going on all the time and massive amounts of process context switching.... So much so that in some configurations over half the 1GB of RAM was being hogged by just passing messages to systems which had yet to be woken by the windows scheduler and clear their backlog of events.

So it was after about six years a new piece of hardware hoved into view, it was a dual core box... Immediately the systems were out of order, things were going wrong, and it was a nightmare.

To be fair, the idea was sound, the implementation not so much and the language used, though flexible incurred (at that time) too high an overhead on the hardware.

Mr B struggled with this, and then he hired a new manager, a sub-manager to himself, to control testing.  Lets call him Mr P... Mr P initially had some good ideas, he used dot to create some process flow information, he tried to control contractor Mr J (who was not a permanent member of staff as he was the only person who knew anything about the message passing) and they were all wrangling to try and make this system work on newer and newer hardware, until one day it had to go onto a wholly other machine....

None of the hardware interaction stuff was abstracted, so it was written all over, but still not abstracted... you could boot the old hardware layer or the new... or even both and make it eat its own tail.

And then there were lay offs... a bunch of the folks working for Mr P went, a bunch of the testers went, and there was a big reshuffle.... I got to work right in front of Mr B... and started to try and bend his ear, to push him in new directions and not rely on Mr J so much... There's a story in that to tell.

However, things became really interesting at the turn of the final year of the whole team being on the project, a new piece of hardware was yet again acquired and the company wanted Phoenix to run on it.

I set about making that happen, and I just ignored Mr P, and Mr J and Mr B.... I made the new hardware version and fixed a huge swathe of problems with booting and data at the same time.  I couldn't disentangle from the death embrace of the message passing and multi-process context switching, but I made things better.*

* One of the things I did was find that the C# file exists check was about 10x slower than doing it as a call to "fstat", so I made a dll with just a call to fstat exposed, loaded that DLL into C# and that made the code massively faster at boot.  I then found all the C# "image.load" functions were really slow, but the equivalent from GDI+ were much quicker, add them to the DLL and voila C# went faster.

So when at the end of Q1 the team was basically told, we're letting you all go, we need one person to keep the project tided over, they picked me... I'd demonstrated a breadth of knowledge, fixed so many problems and hey one can't easily fall out with oneself.

Looking back I think the decision to keep me shocked a lot of people who thought themselves indispensable, but the proof was in the following few months.

Where the fault rate went down from the 20% range, to 18%, then 7% and finally to sub 1% range.  We were on more machines than ever, but we'd cleaned up the code base, the three testers we had went over the systems in a methodical manner.  I instigated a policy of cleaning up the code as I went, removing inter-process communications, reducing the number of vertical slices you had to hop over to get to the functionality you required and I also wrote tooling... Event viewers.... State machine inter-process visualizers, even a GUI designer.

All to turn the human element into a tooled touch of the code, control how it's used.

That didn't ever mean the testers and operations manager (Mr D) was dettered from playing with the system, far from it... But things were better.

The Phoenix project had a legacy though, the structure and controls... The message passing, the installation process, the IPC and the state machine were all a little wonky in their quirky little ways.

So it was in 2014 I restarted my own personal effort to write a new system, at home, alone, in C++.  That was Bluebird and it has a whole other story to tell.