Wednesday, 26 July 2023

My Worst Development Argument Ever

I have had a really excellently interesting sprint with the work I'm currently doing, like you know one of those sprints where you get a real technical tooth into the pie of problems.

That however is neither an interesting story, nor one I can actually tell.

Instead I will fling my mind back a score of years and we'll discuss THE WORST DEVELOPMENT ARGUMENT I HAVE EVER HAD!

The problem started with the platform I was then working on, it was a quite low-powered single core Celeron PC base, running a favour of Windows for Embedded systems (I think it was Windows 2000 Embedded), but it was basically used as a host for a C# application stack which itself was more or less a wrapper around a C hardware driver talking to the various light, button and money handling mechanisms in the machine.


This system of ours then simply would process launch, at the shell level, another child process which was the actual game content.  And there were lots of different games we could fire up.

Our menu and the hardware polling all would back off, it was actually only polling for button and light IO and every second polling for cash changes from money physically being entered; so that was slow to update, but otherwise it was fine and backed off.

Being a Win32 environment it was fairly easy to back off most all the threading and just launch the child process.

Our testing showed that we had a 92/8% CPU split, the system would take on average 8% CPU whilst in this active child mode; it had a few spikes and it had a few troughs, that included windows itself.  Otherwise it was pretty much that most all the CPU was available to the child process.

It was therefore with some scepticism that we had the development manager in the content area wonder over complaining that one of his guys was having issues getting smooth frame rates from the platform.  We accepted the install and ran the child process, indeed we immediately saw the issue.

A popular UK TV show game, with a target circling the board...


The target would stagger and slew around, this was immediately explained to us that the system was so busy the game was being forced to drop frames, the frame rate icon the game had programmed into it went up and down like a pair of kangaroo's in the mating season.  This was the whole argument, our game drops frames your system takes too much time on the CPU.

It has to be pointed out that our manual actually said we only make 75% of the system time available to the game, our demonstrating we were making over 90% available was well within tolerance, and this is the only game showing this kind of issue.  We were therefore skeptical about the claims being made; especially about the high quality and tested nature of their content executable.

After an amount of head scratching we decided to measure how often the "Present" function was being called in DirectX, a little hackery later and we had a measure... Was Present being called consistently and then the system not presenting, or was the game itself changing the length of time a frame took, staggering when it presented?

Yes, the game itself was almost immediately measured to be staggering how often it called Present.  So the question came up "When you move these items are you interpolating between the position and so smoothing where the target is?  Or are you moving them a fixed distance each frame?"

"The frames are a fixed length"... The developer and their manager said.  We had just measured and shown that the frames were changing length, we couldn't look at their code, so we'd had to hack about but we'd shown their code was calling the present with the same staggering it was not a fixed frame rate as they said it should be.

We video recorded (literally with a camera on a tripod) measured the frames on the camera and calculated the stutter.  Then measured with our test harness when present was being called and saw a direct match, when they called present the present happened.  Our conclusion, the only conclusion really, was the content executable needed to be calling present more consistently; or their DirectX present be set up to sync with the screen or something.

The argument then began, they insisted that their game was presenting at a fixed interval, it was fixed to 30 FPS; they refused to turn on VSync, their platform did not allow this setting to be part of the DirectX initialise... It really should have been.  But we had explained the staggering, the delay in present being called, neither were o do with the system all appeared to be in their code.

"But the system is taking 100% CPU"

Yes at this point three member of my department all pulled this face:



It has to be said the manager in charge of this developer is perhaps the worst development manager I've ever met, a man so inept he alienated and lost developers at a prolific rate, who spent £40,000 on migrating to Perforce rather than paying his team and just using git.  Who seemed to think he could pick up modern development techniques by osmosis.

The developer himself?  Well, lets see how he handled all this evidence from the camera footage....


That's right, he lost his shit.

Despite myself, a coworker and our manager all confirming our observations that the hitching and stuttering were coming from the child process itself seeming to go idle and so it calling present at differing intervals he clearly took this to be a personal attack.  He stood in our development lab and basically tore into our system; our whole team took a verbal berating at this juniors gob and his manager relished it (this was one of the earliest examples I saw of this manager relishing in his minions going to town on others instead of his reeling them in).

Rocked by their insistence and being in the unhappy situation of having to prove our innocence our manager asked for us to be able to review their code.  They refused.  There were some issues, because our company had just conjoined with the place this developer worked with, they still saw us as interlopers.

My colleague was known to be somewhat more volatile than myself, so our manager left me on this horrible issue, trying to figure out what was wrong.

Come the Friday, and a little exacerbated after two days, I started to decompile their executable.  It appeared to me that their main loop was miss-behaving, it seemed to be just calling Sleep with a fixed value on top of the work it had carried out each loop.  And sleep takes a variable amount of time.... It was not taking a check of how long their update took and then sleeping for the difference to meet the frame rate target and then it dawned on me.... Did you spot it too?

They were calling SLEEP.

The Win32 API documentation literally says sleep is not a fixed interval, it relinquishes your thread for the remainder of your time slice up to or more than the sleep time given, returning when the windows scheduler next wishes to make your thread active.

It still says similar today, the time slept is not guaranteed.

This client process main function seemed to be calling sleep with a fixed value each pass, it was hardcoded into the executable.  If you sleep for X+Unknown, you are going to see exactly this staggering.  Our demo child application has a busy wait, it never slept, it yielded by passing a sleep of zero (which make it give up its time slice but remain ready, and it would more or less always be rescheduled before the fixed frame rate next time point, plus we interpolated our animation example).

This child process was just proving more and more to not be to specification.

Their manager insisted it was to spec... Which was very frustrating, as he grandly declared "I have read the code, I know it is correct" and "when we run it, we do not see this issue".

It felt like a fundamental miss-understanding, none of the folks on the team I was within were being treated with any respect, nor acknowledging our collective experience and understanding.

I stewed on this all Friday, come the Monday things were getting fractious.  This game had to be out!  It came up in the master development round up that this game was held up by our system.  This started the direct antagonism.

Everything I did, everything other members of the team did, all showed this content application was just staggering and stuttering on it's own volition, by design, by intent it staggered.  It was not our system!

To prove this I therefore set about writing a harness, which would just give the client process the same DLL to load, but they were all stubbed out calls, and it would run WITHOUT our system.  The content could be tested locally and checked.  We got one of the same Celeron PC's, just a flat install of the OS and double clicked my harness.... Sure enough the game staggered about in the exact same way!

I presented a A4 page example and showed how the loop in our example application worked, explaining that Sleep is not a fixed time interval and a busy loop should be used with a yield not a time span anyway.  I recorded this with my harness and their game, I presented both recordings too.

The developer went apoplectic....


He literally shouted at me that Sleep was guaranteed to come back after that amount of time, and when he ran my DLL shim locally it ran really smoothly.

He was right, it did, but he had a much different dual core machine with 4GB of RAM and a graphics card.  Our platform was a single core Celeron 1GB of RAM and built-in Intel graphics shared vram, in short very different.

At this point the director who sat between my manager and this development manager ordered that I be allowed to look at their code.

The worse development argument I have ever had then hit a peak, as I walked over, flanked by my manager and their manager.  I lent over his shoulder and pointed to the sleep function and said that is not going to be a fixed period.

That was all I said, he never let me explain any further, he just went MENTAL!  He started shouting, screaming, and called me a few choice names.  He would not accept all the evidence that his loop was not a fixed length, that it was changing from frame to frame, he just could not figure out that:

{
    UpdateStuff();
    Render();
    Sleep(33);
}

Was not going to always take a fixed amount of time, first of all I pointed out that doing anything and then sleeping like this will be the time the work takes plus at least 33, and then there was no guarantee that the sleep would immediately come back.  The windows scheduler would decide when you can come back after at minimum that amount of time rounded to the nearest platform tick.

His manager immediately backed him up and agreed with him, they both talked down to me.  They insisted loudly and angrily that sleep was fixed and the functions they have took such a trivial amount of time they were not worth measuring.... 

Even my manager backed me up here, of course doing a function call, any function, will take some amount of time and they need to take that into account.

I had just had these two idiots literally shouting at me, whilst I had to stay so calm, it took an icy handful of minutes for them to accept the argument that 33+N is > 33 where N is none zero.  It was just fundamental and they were not having it.

Their code became:

{
    startTime = Now();
    UpdateStuff();
    Render();
    endTime = Now();
    Sleep(33 - (endTime - startTime));
}

Slightly better, but we still saw hitches and stutters, they were far far less frequent now.

This massive drop in frequency I immediately and without changing my argument pointed me to the sleep, as I said the sleep is not a fixed time, it was not going to sleep for X and come immediately back, that's not how Windows worked.

Their argument was that Sleep was fixed, that it was guaranteed to return after X.

They were very loud, very obnoxious and very adamant.

We returned to the developers desk:  "Show me why you think Sleep is fixed".

I expected him to bring up some code, some harness, some proof of his thinking.  Instead he opened Internet Explorer, went to MSDN and showed me the Sleep function documentation.

Sure enough it said "fixed interval".  He was so smug.  So infuriatingly smug.  His manager was ultra smug too.

I reached down, scrolled the mouse and pointed to the screen.... 



He was reading Sleep in the Windows Mobile SDK.  He's right on Windows Mobile sleep is a fixed interval.  However, we're not on Windows Mobile are we.

My manager looked at the screen, I looked at the screen, they looked at the screen.  And immediately the developer called me a horrible name, yup, just straight up called me a name.

I have to admit I didn't react well, fisty-cuffs didn't happen, though the way he erupted out of his seat raging I expected the guy to swing for me.



He could not take it, his manager still argued he was right, so invested in their mistake were they that they could not admit their miss-understanding.  The manager always claiming he only hired the best minds, this guy being quite arrogant and the whole lot of them generally being very dismissive of both myself and the department to which I belonged.

I walked away with my head up high.  My manager stood and pair programmed with both their manager and their developer for maybe twenty minutes and a new version of the content executable quietly appeared without any stutter; even when we ran obrut!

It was a horrible moment in my time with that employer, I remember how the guy never apologised, that development manager never apologised and the game went out without any further delay, but they never received any censure for the episode.

Our department also never shook this kind of effect either, for some reason because their manager had gone to bat for them from the off every following time a performance issue arose we had to prove everything to the Nth degree without ever seeing the other side doing the same.  Very rarely was it ever truly our issue.

I have never forgotten, I have never forgiven.

When you foul up, just admit it, owning it and learning from it is far more wholesome than being uptight and obtuse.

Friday, 14 July 2023

Publishing Tribulations

Okay so I will start here...


Now I have that out of my system, what am I talking about?  I am talking about English.

As you may notice I am English, I speak English, I write in English I am fluent in English.

I recently sent a piece literature for publishing, I've self published before, this time it was going to a publisher.... Their reply:

Sorry to hear you are unable to response in person for your trities will not meet our publishing guideline.  Please check and correct spelling, punctuation and parographing, formatting and font.

Very seriously this was the reply I was sent.

I have withdrawn my manuscript for consideration with them, sending it to penguin... No no the publisher, just a penguin, it'll be able to give me a more coherent reply.

Sunday, 25 June 2023

Soooo... About the Ballad of Buster Scruggs....

It has taken me a very long time to watch this film in a single sitting, and a very long time tolerating it, so much so that upon completion and having given it some years, I immediately searched the internet for people of a like mind as me; that being that it's a load of rubbish.

There are a few out there, mostly on hidden secret dark little corners; of which clearly this is another example.

The open chorus of plaudits for this film, if we can call it that, all fall into two broad but terrifying categories.  The first are Cohen brother fans, they won't be deterred and I can agree with them on some of the points they raise.  It looks great, the cinematography and sets are for the most part great and it has call backs to great westerns.... Like the name of the Bank.... Little nuggets of gold itself.  And individually the performances are great.


The second group are those voices which really annoy me, they say "you have to be smarter" to get the joke... To read between the lines.... To understand the history.... These people are the worst, not the least as their attitude is very insulting, which it very much is, and condescending.  But really we are not dumb, we are not being obtuse and really should we need a social education background in order to comprehend some of these stories?  No, not really.

So in order, and I am going to go from memory here, we have:

Buster the gun slinger story.
The Bank robbery.
The Gold Prospector.
The limbless Thespian.
The Wagon Train.
The Stage Coach.

The weakest is the Stage Coach, just extremely weak, and unlike the others it looks the worst.  And it was really hard to watch, really quite boring and ultimately had no point.  You can tell me it did have a point, but watch it again... Did it?  Really?

The Wagon Train, it had something, there was something there.... I was even hoping that Billy/William and Alice would tie the knot but then the other Wagon Guide would fly into a jealous rage, like there was some homosexual tension or active romance the younger man was basically calling off; just something which would have explained the story, nothing did... Sure I watched the story, but it was a pointless story.

The Bank Robbery had a very strong start, you had the explanation of the previous robber being kept alive for the Marshal for three weeks and ending up doing hard labour, instead we cut to James Franco strung up... Twice... Just... Erg, there was no fabric, the two halves of it made no connection to one another save for his horse not listening to him, that was the only link between front and back.

We're half way through this production, I can already see in the last story it was simply rushed and done very much to a budget, but was pointless, so we're not far from useless.

Again the Wagon Train, slow burn start, quite interesting middle, the tension over the marriage... But then, just as you think... "Oh the lover of Billy saved Alice, now he's properly torn between the two" or something... boom... literally, killed the plot.

The title carrier, Buster Scruggs, well that was my favourite of the stories (closely followed by the Prospector) I thought that was going to be the whole style of the film, heck it is its very title.  But then they shot that down fast, and the looney toons soul rising to heaven scene.... It reminded me of the laughing hyena's at the end of Who Frames Roger Rabbit.... And just was no in keeping, it was so jarring for the whole rest of the film.  And it ultimately made no sense, like if we saw the spirits of the other gun slingers he fought, maybe.... There might have been an Undead PD thing, or a Frighteners Wild West style thing... But no, no pay off, just... a song.

This leaves the Prospector to save the film, and I have to be honest, it was slow, but my favourite story.... His throwing away ANY gold though, even little flakes in the first few pans... Just so they can say "keepers, we're getting to keepers" is pointlessly annoying, and really flies in the face of the "you need to know the history kind"... really, know the history, ever seen anyone knowingly discard gold?  But it had an interesting parable with an echo of echo message, he entered a virgin, un touched wild, dug holes, shared with the owl, nature reclaiming it almost immediately with the deer at the end... Even though they were just holes he made little impact but got what he wanted, which is a good way to think.

Rather than watch, ride on someone else's efforts and steal from nature, from the Prospector.... It's a moral tale and the only one to pick out of the bunch.

If either of the brothers were to ever read this meager gaggle of gripes, I'd say to them to loose the yes men, sometimes things aren't very good, you done goofed boys.

Sunday, 11 June 2023

React Physics 3D : First Impressions

With my home engine coming to the state of being able to visually represent a world and move about within it, I've decided to set about sorting out a physics representation.

Very much as I did with the mathematics where I set about writing all my own code and then adopting a library; in order to best learn the process from first principles I also set about writing my own physics engine and visualizer (the latter using the much more familiar to me fixed function pipeline of DirectX 9 - Yes it still has a use in 2023!)

The physics implementation I went with was a simple rigid body test for cuboids and spheres.  Then a ray cast.  At which point I set about seeking a library as it's a huge topic I didn't want to sink too much time into.

I settled, after about a fortnight of reading and trying on React Physics 3D.

My initial impressions are that it is easy to set up, supports CMake and just worked; though I'm yet to fully get to grips with it, I am happy I have my world and simulation set up and working.

I've also happily separated my game objects from my visual objects with the use of my Ecs and I'm able to do the same, by registering a new Ecs update system with my entity list and simply piping the commands to create and destroy physics objects per frame.

The update pumping of time matches my engine instantly, so I had a lot of wins.

However, I am not building in a permissive manner, I have maximum warnings as errors, I have strictness on and permissive off; I also remove all compiler specific extensions, so I am using only C++ in it's rawest, most portable form.

Immediately, even though I've compiled React separately and then just link to the library statically, I have a bunch of the React headers giving me errors; notably map, BroadPhaseSystem and DefaultLogger.

The first two are the same sort of issue, which just surprised me, there's a constexpr uint64 set to minus one (so it's max) an unsigned value set to a signed negative to under roll it... Well it's quite bad practice:

static constexpr uint64 INVALID_INDEX = -1;

And I immediately updated this to:

static constexpr uint64 INVALID_INDEX { std::numeric_limits<uint64_t>::max() };

I may feed this back to the author, he most likely knows, maybe there's even a reason for it, personally I'd always stick to numeric limits and max in this case though, not least as it is constexpr and not a hack ;) 

The other one was a lambda expression in a call to std::transform for making all logging strings lower case, the code simply used ::tolower.  However, it was not type safe, it was converting int to char, but that's not explicit and so I changed it over to read:

std::string toLowerCase(const std::string& text) {
                std::string textLower{ text };
                std::transform(textLower.begin(), textLower.end(), textLower.begin(), [](const char& character) -> char { return static_cast<char>(std::tolower(character)); });
                return textLower;
            }

The lambda here is hard to read, so lets split it out:

[](const char& character) -> char 
{
    return static_cast<char>(std::tolower(character));
}

We take in a character, return a character, explicitly, so we have to cast the int return of std::tolower... simple really, but hard to read.  It compiles down well; but just using "::tolower" well that's a decay to int and you have to be permissive in your type exchange... I don't like that, express yourself in your code type correctly, and it'll be type safe.

Friday, 2 June 2023

Thursday, 25 May 2023

Just Stand It Up: About Premature Pessimization

Engineers often talk about premature optimization, but today I'm going to just talk briefly about the opposite, premature pessimization.

I currently work on a very large code base, it has been developed over four years from scratch.  One of the first things performed were a series of investigations into "best performing" data structures, such as maps, lists and so forth.

Now of course one has to totally accept one can optimize any data structure that little bit more for a specific use case.  One also accepts that when in C++ the standard library lends itself to being replaced bu defining standard operators, iterators and the algorithms going with all this use those standard exposed APIs, so you can implement your own.

I just want you to stop though and think... Do you want to?

Too early in a project and you can start to introduce bloat, either in terms of slight differences in the optimized cases, the code from whatever third party "best" you picked and even from your build chain, as you are bringing in dependencies on someone else.

The standard library doesn't do any of this, its dependencies are guaranteed by the ABI.

So why not just use standard map, or standard vector and standard string, or standard formatting?

Quite often I'm finding it is due to premature pessimization, that developers voice who cries out about some issue they had, either when some technology was new and emerging or late in an earlier project's life where they had to optimize for that specific case I mention, where the standard version did prove itself to be a detriment.

These engineers carry with them the experience, sometimes scars, from such exposure to edge cases and bugs they had to quash.  Rightly and understandably they do not want to experience these self same issues again; their minds therefore are almost averted from just standing it up with the standard version.  They immediately seek and even proactively nay-say the standard versions in favour of domain specific "best" versions.

This is in my opinion the very definition of premature pessimization, the standard library is wonderful, diverse and full of very well tested code and will have nearly zero overhead in adding and using to your project in C++.

I would therefore coach any developer with such mental anguish over just using the standard library to simply stand it up, just get across that line of things both building and running, then extend it to remain maintainable.  And finally as you think you're getting close to stable, well then you can expend more time looking at, profiling, and understanding the edge cases.

Sunday, 23 April 2023

Missing Emergency Alert

Just 30 minutes ago the UK was meant to test a nationwide all handsets Emergency Alert system... They did, my wife received the message and we saw someone at the Crucible in Sheffield at the Snooker also get the alert.

But I didn't get it.... Precious few others I know seem to have received it, in total I can say two, the strange on the Tele and the Wife, that's it... No-one else has mentioned getting int, quite the opposite, lots of people interested in the technology didn't receive the message.

So what gives?

Has there been some technical glitch?

Did handsets turn out to not be compatible?

Did some spam protection in the networks stop the message?

It is curious, and I'd love to find out more.