Monday 4 March 2024

Tech Tribulations #2: There's a New System in Town


Time for another story from my tech history, this time I point my torch of memory at one of the first larger systems I worked on from start to finish (nearly 10 years).

I think I can say the product name of this particular system, it was called Phoenix and it wasn't a bad idea, to be honest, as with nearly every project I've ever worked on we needed more documentation and design; those to continuously neglected facetes of software engineering.  But other than that, the initial design stood up.

Before we dive into that though, we need to understand the reason Phoenix was thrown onto a whiteboard and born.  What was the direct predecessor?

Yes, there was one, this is not a bluesky thinking system, Phoenix was to directly replace a system which had itself been engineered over the prior two years and not really lived up to expectations.

I shan't name that system, but anyone who knows me and knows Phoenix may know the code name it was given.  It was a hybrid of two languages a C wrapper around the physical hardware bound with JNI to a java core, which itself used the ODBC driver to talk to a Marathon database backend.

The idea being that the transactional nature of the database allowed for the representation of perfect state, and in any error it was simply a case of rolling the transaction off and so we had RORI behaviour in a very complex system.

The problem?

It was SSSSLLLLOOOOWWWWWW.

The host machine was (and I think I'm right here) a Pentium II class machine (it may have been a Pentium III later) with 16MB of RAM (and I believe it went up to 64MB by the end).  Am I talking about early 2005 here, we were working with legacy machines in the field which were being refitted from a system itself all written in C but not well understood.

This new modular java, database, transactional system was envisioned to be the future!

The problem?  Still it was SLLOOOOOOWWWWWW......

I've said that twice now, so for effect lets skip forward to about 2006, two years in and this database java monstrosity is not doing well, it doesn't perform well, the team has literally stopped even trying to work in agile scrum manner; the lead developer didn't take well to critique amongst other issues (like it's slow, have I said that three times now?)

Happily however there were two separate development machines, the one on the old old system and a whole other on the java junk.  So it was in the C group I was called into a meeting with my manager, the manager of the whole department.

He had a punch bag doll in front of a whiteboard and asked me to close the door.  Furtively expecting my P45 (to be fired) he asked me to take a look at the board... he had asked two other engineers to take a look through the day, so he was canvassing for opinion.

In the dry wipe ink was a block diagram, eight boxes, a hardware module to talk to hardware, a module to account, a module to host the menu, a module to talk to the content program, and so on, one for logging, one for sending reports back to HQ... These simple systems.

Each would talk to one another with a message passing mechanism, which was a line from each of these eight boxes.  Little did I know then that this message passing system would be the biggest part of my debugging life for the next ten years; that however is another story.

Right there and then this new system seemed like a much better proposition.

The manager liked the feedback from the various folks he had to talk to, and it was decided upon.  He would work on the platform, starting up the system, securing it with a manifest and preventing unauthorized execution of random code and content.

I would be working on the main attract module, this is the main menu from which content is categorized, and launched.

The content would talk to the Game Interface module, which was owned by a guy called Darren.

The accounting module was handled by the previous lead on the java system, and later a guy called Conrad.  Conrad at first though would be doing some other systems.

And there was a smattering of other work, like a logger module, and a build stack and al-sorts.

Work began, but not before the lead on the then current system had to be told her baby was being canned.  She and her main ally were duly summoned, shown the block diagram and where they would be working.  She did not take it well.

As she left the managers office, he was heard to exclaim and his stapler threw across the space and nailed a window barrier, sticking into it with a vibrating twang.

She loved her system, she did not like Phoenix, not least as it's name was a direct reference to something rising from the ashes, her system being declared to have self-combusted in failure.