It's a C++ STL Rant : Constness and Vector Erase

I believe I am right and correct to be seethingly annoyed that this code does not compile...

#include <vector>

#include <iostream>

#include <algorithm>

struct Test

{

    const float value;

    int changable;

};

int main()

{

    std::vector<Test> items;

    items.push_back({42.5f, 0});

    items.push_back({1.23f, 1});

    items.erase(items.begin());

}

It just really annoys me this won't compile, do you know why?  Don't cheat, just have a look why does this PERFECTLY REASONABLE AND ACTUALLY I THINK CORRECT CODE NOT COMPILE?

Yes, because of the ISO Standard, and it's really annoying.

ISO/IEC 14882:2003 23.1 [lib.container.requirements] / 3:

The type of objects stored in these components must meet the requirements of CopyConstructible types (20.1.3), and the additional requirements of Assignable types.

The compiler could have been told this far more simply if the std::vector had an earlier and cleaner static_assert that the type is copy constructible before it got into the weeds of the real message we get.

For can't compile because it seems the standard implementation of erase results in a bunch of move semantics in the vector, which then devolve into copies.  And to my simple monkey brain it's all just babbling horrible template junk that comes out the compiler as well.

You can of course move a struct of type Test perfectly well, or at least mark it for move (cast it thus):

#include <vector>

#include <iostream>

#include <algorithm>

struct Test

{

    const float value;

    int changable;

};

int main()

{

    std::vector<Test> items;

    items.push_back({42.5f, 0});

    items.push_back({1.23f, 1});

    Test bar { 123.456f, 2};

    items.push_back(std::move(bar));    

}

This compiles and runs just fine!

But it's not copy assignable.  Let us take a look at the compiler spew:

In file included from <source>:1:

In file included from /opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/vector:60:

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:427:4: error: static assertion failed due to requirement 'is_move_assignable<Test>::value': type must be assignable

          static_assert( __assignable::value, "type must be assignable" );

          ^              ~~~~~~~~~~~~~~~~~~~

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:495:22: note: in instantiation of function template specialization 'std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<Test>' requested here

                              _Category>::__copy_m(__first, __last, __result);

                                          ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:522:19: note: in instantiation of function template specialization 'std::__copy_move_a2<true, Test *, Test *>' requested here

    { return std::__copy_move_a2<_IsMove>(__first, __last, __result); }

                  ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:530:8: note: in instantiation of function template specialization 'std::__copy_move_a1<true, Test *, Test *>' requested here

                std::__copy_move_a1<_IsMove>(std::__niter_base(__first),

                     ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:652:19: note: in instantiation of function template specialization 'std::__copy_move_a<true, __gnu_cxx::__normal_iterator<Test *, std::vector<Test>>, __gnu_cxx::__normal_iterator<Test *, std::vector<Test>>>' requested here

      return std::__copy_move_a<true>(std::__miter_base(__first),

                  ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/vector.tcc:179:2: note: in instantiation of function template specialization 'std::move<__gnu_cxx::__normal_iterator<Test *, std::vector<Test>>, __gnu_cxx::__normal_iterator<Test *, std::vector<Test>>>' requested here

        _GLIBCXX_MOVE3(__position + 1, end(), __position);

        ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:656:44: note: expanded from macro '_GLIBCXX_MOVE3'

#define _GLIBCXX_MOVE3(_Tp, _Up, _Vp) std::move(_Tp, _Up, _Vp)

                                           ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_vector.h:1530:16: note: in instantiation of member function 'std::vector<Test>::_M_erase' requested here

      { return _M_erase(begin() + (__position - cbegin())); }

               ^

<source>:20:11: note: in instantiation of member function 'std::vector<Test>::erase' requested here

    items.erase(items.begin());

          ^

1 error generated.

Compiler returned: 1

We read this from the bottom up, first erase needs to call "_M_erase(begin() + (__position - cbegin())) what on earth?  Well it appears this is trying to perform a copy-move of the rest of the vector; the rest of the stack all the way to the top shows us various iterator types to the top two messages where we have __copy_m and then a compile-time static assert that the type must be assignable.

With the intentionally const member we can't compile we therefore have two options, first we make the member non-constant.  This isn't very practical because the type is derived from a bunch of runtime values, they're not known upfront and can be different for each instance of the struct.

The second option therefore is even less appealing and looks like this hideous mess:

#include <vector>

#include <iostream>

#include <algorithm>

struct Test

{

    const float value;

    int changable;

    Test& operator=(Test pOther)

    {

        float& mutableV { const_cast<float&>(this->value) };

        mutableV = pOther.value;

        this->changable = pOther.changable;

        return *this;

    }

};

int main()

{

    std::vector<Test> items;

    items.push_back({42.5f, 0});

    items.push_back({1.23f, 1});

    Test bar { 123.456f, 2};

    items.push_back(std::move(bar));

    items.erase(items.begin());

}

We define this assignment operator performing this const reference magic and just.... ergh, don't do this folks, just don't.  Not only is this an excellent lesson in const member flagging in STL and it's pitfalls.  But then it's also a lesson that const doesn't really mean const.  AND it's a lesson on whether you want to keep a constness or nor given such awful hacks to get around it.

I could even go further than this, the operator= could be made private and the STL type "std::vector<Test>" could be made a friend, or the __copy_m could be made a friend; though in that last case the MSVC implementation of the STL will be different, so we'll have to work around platform differences too.

No, the most annoying thing here is picking open quite why __copy_m wants to end up assigning.  I am still digging into why, but oh boy am I annoyed by this one.  For in a module I unironically had to repeat at University solved this in 1996.... I would use a linked list, and simply pluck out the value... or I would skip to the index of the item to remove and copy those after (if they were host memory contiguous then memcpy speed achieved) and just write over and this erase is achieved.

To just see this wanting to copy assign at the element level is just hideous, and I hate it, and I with the ISO standard would change in this regard,

Story Time - Fantastic Rack Mount Mistakes - Part 33 and 1/3

It has been a very long time since my last story time, and so this one comes to you with an unknown number, I am however well beyond any possible NDA or secrets for this one, so I can finally share (it's been 20 years yesterday).

I had a job in a data center, nothing too special, but I loved it.  The loud room, the air conditioning, the rack upon rack of machine, I absolutely loved it.  We're talking 2002 here, these were the 2U high style of machine I use a lot today, the odd 4U or 8U JBOD box too, loved them all.

I had all my wiring terminated and was beautifully dressed, I wasn't in charge of the whole place, just one isle, but I took my time whenever there was an issue to run new cable bundles beautifully dressed along the cabinet runners into each cabinet top, seven cabinets to a row, two face to face on the cold row and then the back of these into the hot, I had a hot row run by a lovely guy called Jerod next to me, he was French I think, lovely guy (shout out to Jerod if you're reading!).

The other side I had the work bench, a coveted space, between the hot and cold isles were thick plastic strip drapes, like you'd get in a meat packing plant.

We would patrol our isle, identifying any issues or disks showing physically bad, schedule them in for late night fix ups and then to into the communal office where we each hot desked.

Each of us had a laptop and would sit and connect into the monitoring suite remotely, with a big screen we connected via VGA to our laptops and dozens of spare keyboards we could quite often spend the night shifts doing nothing but playing games; Medieval Total War was the order of the day for me at the time; some of you know of my origins in the "game industry" take me back to hacking about and modifying MTW (Sorry Tom, whom I now work with, for fishing about breaking your baby).

Anyway, about three months into this job another tech on the day shift came to me about Isle 4, the far end of the room, he was getting drop outs at intervals during the day, but none at night, he was trying to figure out what might be causing it.

We took a look at his logs and sure enough on his disks he had 250-700ms delays for seek times on his platters, he also had a higher than average fault rate on disks, apparently he was also rebuilding raid arrays a lot.

He asked us in the night team to watch over the kit live; which we duely did.... Quiet as a mouse, nothing not an issue ever in about three months of monitoring.

Meanwhile in day light he was tearing his hair out; I've got to admit to genuinely not remembering his name, but he had long greasy black hair and was from Northampton which he told me more than his name... "Hello I'm <forgetable>, I live in Northampton".

But forgettable as he was I can't forget the day he handed that isle over to me.

Stumped as to what was going on, and frankly pulling rank, he shuffled Jerod down to Isle 3, himself into Isle 1 (my isle) and all new machines were coming into isle 2, which mr senior with disk failures was going to be setting up.

This really irked me, my isle was by far the best presented, when sales wanted to show new accounts around they showed them my isle and they all laughed over the name of the cabinets; we couldn't name the machines, most of them belongs to customers, but we could sticker and name the cabinets themselves, so they all got Bond Theme names.... Jaws, Octopussy, Moneypenny, Q and M all had simple servers, processors more or less and my two JBOD cabinets were Sean and Roger.  Stencilled on in lovely white enamel paint.

So I was hoofed off to isle 4, dark, at the back, no work bench... and immediately set about making it my own, I redressed the cable bundles (which annoyed Mr Senior, though he loved my already done ones, my dressing his was seen as an overt declaration of war).  It's so simple too, cable tie around five, cable tie through the middle to break them into three and two sets then pull.... Today you can get cable combs and I find them so sexy.

Anyway, dressed to impress what was not impressing was the fault rate, and instead of it being the accepted norm; as it had been for Mr Senior for months; it was now a big deal because I was in charge of the isle.

I set about trying to work it out, I could not figure it out.

At night, when I was there it was all fine, the day guys saw nothing; they pretty much just tended the machines remotely, no maintenance happened between 7am and 7pm so it was up to the night guys like me to swap disks, rebuild arrays, fix cables and install new machines.

So after two weeks with the log I got showing all these faults, like what was causing a disk which had worked all night and most of the day suddenly to show 500+ milliseconds seek!  It made no sense.

Anyway, 2002 was coming to a close, December 21st rolled around and I offered to cover a couple of the guys shifts, they had kids it was nearly Christmas and we worked Christmas Eve and half Christmas Day, one guy on on guy off for 6 hours at a time those two days, I took two shifts back to back.

Alone in the office, I watched the monitors and played games.

No spikes.... What gives, no spikes on the DAY I am actually here.

Hold on, I'm here, but no-one else is.

Could it be human interference?  I checked electrical circuits for noise, I checked lights, I checked if the air conditioning was affected outside and in.  I checked everything, no signs, no peaks, no noise.

HANG ON.  NO NOISE!

Spinning disks can be affected by loud noises, specifically by vibrations, I'd seen this demonstrated when doing my Compaq certification training.

There was no noise, could it just be vibration?

We were on the first floor (for anyone in the US, this is the floor above the one on the ground) so we are one storey up.  There was the main reception below center of the room, to the right where isle 1 is would be a hall way void with offices leading off it, under isle 4 would be a toilet and a changing room.

I went down, put the shower on, came back up.... Nope.

Then I took my laptop with me, no wifi, but I could plug it into various office ethernet ports around the place as no-one was in.

There I am Christmas Day, banging doors, flicking lights, flushing loos and then pouncing on my laptop to see if it affected any of the disk activity I was artificially running up.

Nothing.

Defeated I logged my time, handed over to Jerod and went to have my Christmas dinner.  I had 12 hours to think of something for Boxing day.

I walked into Mr Senior asking why there was CCTV of me "Dashing about with my laptop in random offices".

I just said I was trying to check light circuits for his disk issues; he made it abundantly clear they were my disk issues and left me to it.

Boxing day, the sales had started, the building was at the corner of a large commercial estate, there being a newly built Ikea across the road as the crowds rolled in and their stock levels fell they would be due a delivery soon.

I still poked around the office and the eureka moment came on the 27th December 2002.  For a large lorry was rumbling past the office, I had to wait for it to pass, a massive blue IKEA clad lorry.  I went into the office, logged into the monitoring and sure enough there was a trace of a large disk issue.  Times 4 minutes before; when that lorry went past!

I didn't hear it, I didn't feel it... But had the disks?

I waited and watched when a few lorries passed during the fairly quiet week between Christmas and New Years, nearly every heavy lorry going back resulted in some affect on the disks, vibration was being carried into the ground and I guess up through the building.  I will be honest, I could not tell.  But every spike I saw was timed with a lorry.  In fact I soon let Jerod in on my idea and plan to fix it and so he watched and I monitored and when he came in I could tell him when a lorry has passed!

I was not about to shout about this to Mr Senior, instead I set about fixing it.

I ordered a mat of 1 inch thick rubber, the stuff you mount washing machines on in your kitchen.  I already knew the racks pretty well, they were bolted at each foot, I'd need a torque wrench to unbolt them and I could use two of the hydraulic scissor platform lift trolleys we used to move machines about to lift the rack ever so slightly.

I didn't want to do this alone, so Jerod was roped in with the promise of a take away pizza from the glorious; but long gone; parlour we loved.

Mr Senior handed over to me and Jerod that night, it was not uncommon for two of us to be on at night, especially if there was work to do fitting something out.  It was normal to have two people when we were lifting machines too.  But Mr Senior would have been apoplectic if he'd know what us two kids were about to do.

I unscrewed the first rack from the floor, swept the dust out and used a wooden baton to bridge the lip to the metal of the scissor lift and I cranked it.... The cabinet moved, I nearly wet myself as it looked like it was going to topple, then Jerod jacked his side up on the other side and it came level, with an inch to spare I slipped in two of the pads.   He slipped in his two and at a shout over the loud AC we lowered.

I then used a phillips head screw driver to puncture the rubber and a knife to dig a bit out through each bolt hold and fastened the bolt back through, not too tight, just tight enough.

It was sweaty work in the hot isle as I was, but we did the first three of seven that night.

I went home and decided to pre cut the squares and use a drill to cut the middle out of them,the next night all seven were done.

Our disk fault rate when to zero.

That was my last time as an IT minion; I went back to programming soon after, and Mr Senior never was told what we did... His training should have told him.

What did Don do?

First things first, we're going back to the latter half of the 90's here; and I've mentioned Don before in one of my older posts, skidding into hero mode.

Don was a programmer, he was kept in a little plexiglass bubble of his own and spent all day in what looked like vi (certainly a terminal based text environment) writing what I think was C for the NCR minicomputer running Sco Unix I mentioned before.

Whether this is actually true, I don't really know, for he spent a lot of time sitting with his hands crossed over his head waiting; at the time I didn't know what he was waiting for, at home I had a Pentium 100Mhz rocking 16mb of RAM and running Borland Turbo C++ for DOS I had a lovely little progress window showing me it was compiling; I had not met the glory of GNU yet.

So Don would stare at a massive (like 19 inch) monitor, he had a Wyse VT100 next to him too, and he would not stare at that very often.  He would also make copious notes on paper and seemed to program very deliberately, looking and reading through the information at hand.

He did this, almost without interacting with anyone else for the whole year I worked with him; looking back on it I wished I had been more proactive and just asked him more about it, as he was clearly programming C around a Unix system, which is a) uber cool and b) uber cool.

It always struck me quite how long he thought about and maybe designed anything before acting.  You could say this was because he was mulling and considering the problem, he was being a good engineer, with forethought and wanted to get things right?

No, see I think he was really just in fear of how long the compile would take...

Just like me with the project I'm working on twenty five years later.

Confessions of a Linux User

I've never understood this one, but I've felt bad for it for a while now, so I'm going to come clean.... At my last employer they had a very strict network/internet traffic policy, like everything was locked down.  I had a few issues with this because using Windows and Chrome it didn't just block the at the DNS or URL returned level, but it actively broke open HTTPS packets and inspected their contents.

YEAH I KNOW RIGHT?

This was meant to be secure, but it clearly wasn't and I asked about what was opening these packets and inspecting the contents and bless 'em the IT department said it's not... It clearly was.

Because, slip over to a none-Microsoft operating system, even one running in a VM on the self same windows host and it could get out to the internet and retrieve whatever it liked.  So it wasn't just blocking the URL it was inspecting the actual traffic, the actual supposedly secure traffic wasn't secure.

I'm pretty sure this would have proven illegal; just like Scunthorpe Council was found reading its employee's email.

Either way it wasn't that they were doing this that actually bothered me, it was that when asked they said they weren't.  Which meant one of two things, either a) they were and were lying or b) something else was and they didn't know.  I believe it was a case of option A.

For you see besides watching copious amounts of YouTube and listening to even more music, which no-one else could I saw the darker side of this power I had through Linux.

You see the packets were being broken over to inspect their contents because IT wanted to filter certain search terms or words, you know like porn or weapons, nasties.  HTTPS connections can't be inspected (at least between your browser and the receiving/signed server they shouldn't be) so the software on the Microsoft pipe, or Windows itself, was compromised in my point of view.

That's not to say I was looking for tits, but I could have if I wanted to.... Tux!

I love Wales, and I like Scotland..... So IndyRefs?

I'm British.... If we want to delve further into that I was born in England and can trace an 8th of myself to very strong Irish Ancestry, the rest is a bit of Norfolk and a lot of Nottinghamshire.

As a kid I spent time holidaying in North Wales; and of late I've spent a bit of time holidaying on Anglesey.  I like Wales a lot.

I also like Scotland, I had a lovely friend who is Scottish, I've many acquaintances who are Scottish, and I toured Scotland once with notable stops in Dundee and Edinburgh.

Lately the wife and I have even contemplated moving to either country.

However, the thoughts of moving to Scotland were tempered with all this me-gudgeon about IndyRef2.  Now, I thought the matter was settled in the first, it was a close vote, I think every Scottish National voted; and they were out voted by everyone else; there were even a bunch of apathetic people who didn't even bother to vote and still the nationalists were out numbered.

And to be perfectly honest, I think recent times have taught us you're stronger together.  Doubly (or is that quadrupley?) so in terms of the UK.

So hearing all this hoo-haa about the possibility of another referendum and wanting to put things into perspective I took a look at the Scottish Governments own publications; they make for interesting reading.

And I of course focussed on the economy and how Scotland may fair as an independent unit, my question came down to one of Revenue versus expenditure, and to be perfectly honest if I were a Scot asking for independence I'd be scared stiff.

As an Englishman this is perhaps easy for me to say, and feel free to dismiss me; but I'm talking about the FOI-202200277763 publication of the Scottish Government; where it clearly states that "Around 40 per cent of the Scottish Budget 2020-2021 was funded from revenue raising powers devolved to the Scottish Parliament."

40%? Where does the rest of the 60% to keep the country afloat arrive in the coffers from?  "a block grant from the UK Government".

Right, so a lot of money goes into Scotland which it itself can't afford to provide, and I've taken a look, no-one; not one single party calling for IndyRef2 has an answer to this problem.  Indeed they don't ask it, and I find that so very worrying.

I'd even proffer my own opinion that is I, or anyone in Whitehall, asked the question they would have a decorative Shortbread tin stuffed down their gullet faster than you can say "Nicola Sturgeon".

So how do these politicians pulling on the strings of Nationalist Sympathy think the money will come from?  Honestly, I can only believe they think the UK would continue to stump up the cash; forgetting there would be no more UK.

The United Kingdom would cease to exist.

There would be Scotland and then the rest of the Kingdom; England, Wales and Northern Ireland.

Assuming the latter did not continue to fund Scotland, which I believe they would in some way, but assuming that "we" didn't what would the Scottish Expenditure Portfolio become?

Well, I believe Health and Sport would take a hit, particularly Sport.   Quite what the split is between the two is not made clear in 202200277763.  But I think sport would go.

Communities and Local Government is the next big ticket item; well there would not be any "Local" in Government, there would just be the Scottish Government; but does that necessarily spell a saving?  I don't think so, for there would suddenly be a requirement for institutions to facilitate the functions of state.  For example, the Bank of Scotland would have to issue actual currency, Passports, Drivers licensing and control; so many things assumed to cost zero to Scotland today as they're provided services of the state would have to migrate and be picked up by Scotland.

Indeed in the current portfolio "Government Business and Constitutional Relations" shows as zero; because there is none, that cost would sky rocket; more so in a hostile departure from the Acts of Union.

Similarly the "Audit Scotland" would rise, Scottish Parliament too... So many things.

And I can only see the people of Scotland suffering, or the people remaining in the Union being forced to bail out the situation; so I don't get it.

Personally I do not get it, I do not see any need for England, or Wales, to depart the Union, and neither can I see this for Scotland.  It's almost an emotional reaction from some Scots, that they want to be independent, but asking them quite what that means, it means they want the status quo AND freedom; but looking at the books that can't happen.

Whatever your thoughts mine are quite sad, I'm personally strongly against moving to Scotland with this question poised as it is.  I would only move to Scotland if it were part of the Union anyway.

My References:

https://www.gov.scot/publications/foi-202200277763/

https://www.gov.scot/policies/government-finance/setting-the-scottish-budget/

https://yourviews.parliament.scot/finance/scotland-public-finances-23-24/

https://fraserofallander.org/scottish-budget-guide/

https://ifs.org.uk/articles/scotlands-underlying-public-finances-are-improving-oil-and-gas-revenues-rise-long-term

Viva la Treeland!

I would like to call upon the international community to welcome and ratify my call for a vote of independence in the Russian Far East; you know, just randomly, I'm going to call for a vote for some part of someone else's backyard to become a country in it's own right...

Because seemingly one can do such absurdly strange things and the UN not bat an eye lid....

I also wish the local people to know I will be naming their new country, which I shall become a non-executive head of state for... erm.... Treeland.

Yes, TREELAND!!!

Take that Putin, you knobber.

Home Engine - Texture Mapping & Camera Update

I wish I had found more time this week to work on my home game engine, but I've been stuck doing DIY like things; including sorting out the installation of a new gas boiler after going through every possible green energy alternative and finding none for less than £15,000.....

So, what did I get done in code land?  Well, I added texture mapping to my home engine, this was a major step, I also sacked off the blender exporter which I wrote in Python to expose the blender meshes and loops into my own format.  I've simply reworked that hand written format to be less verbose and simply be simple.

I've improved the camera controls added previously, but still have things I want to try there, but for the feel of the game I have in mind for my first pass "project" in the engine this will suffice.

Ray Casting is next on my list, and to that end I started to write my own little Physics solution providers, the first thing added was simply to collide the camera point with the ground plane and bounce it back up and point it slightly skywards again; this both stopped the camera going out of bounds, but also hides the near clipping plane cutting off the ground as the camera gets too close to the ground geometry.

Ground geometry is perhaps one of the higher priority things I want, a bit of diversity in what I'm seeing, but also a heightmap and collision solution against that for the camera.

I also plan to put the Tank & Turret problem solution I have back in, it just needs a clean pair of new models.

And that's the big word for the next couple of hours I find spare... CLEAN UP... I have a lot of cruft in the code, from hard coded models in arrays, to unused vertex descriptors to simply some areas where I've not followed my own coding standards (I found two of these very frustrating in my Vulkan API to transfer images from staging buffers to actual images & views on the device earlier).

I'm not talking about a full pass through the whole vulkan demo, more just the demo I have.

Ultimately I will need to stop writing this demo and start to back port all the code into a Vulkan Renderer Implementation proper (like the DirectX11 one I have), but that's going to take a little more head scratching.

My Physics solver is very rudimentary too, points within boxes, checking which side of a plane they're on or being parallel to.  I plan to add rays and spheres soon, then boxes of course, and I'll have a very basic rigid body and intersection test system; I'm sure the depth buffer will come into this, but for now I'm keeping it all on the CPU.

In Memorium HM Queen Elizabeth II

Rest in Peace Her Majesty Queen Elizabeth II

God Save The King

Home Engine - Bank Holiday Camera Controls & More Models

After the silence earlier in the year I'm so happy to be blogging more often, and I hope you are all enjoying it too; I can see some regulars popping by and a few new faces in the stats list; but still far too many of you are totally anonymous, drop me a comment!

Anyway, what I have I been up to?

Well, under the hood I've totally revamped how the uniform buffer objects are passed to the rendering operation in the vertex shader.  I've also moved the view and projection matrix calculations into a nice camera system class, so they're contained and sensible to use.

As a result I've then added some key bindings to move forward, strafe and rise/drop the camera on the plane of x-z.  Likewise I've added a free mouse mode to let me move the mouse and fly over the terrain.

Speaking of terrain I've added a checker board to show me the X-Z plane at about -0.5 Y.

I also noticed a winding order in my Blender exporter tool, I've not fixed that in the python, just patched it in my loader and created a few more models to start building out a play space.

Technology wise I need to make the camera more usable - a full third person camera basically.  Then I would like an orbit cam, which moving in the strafe would move it around the locked object.

Then I'm going to think about texturing and lighting.

The solid coloured aesthetic is intentional though, so anything more fancy is a low priority for me.

I am feeling much more comfortable in Vulkan though; and of course enjoyed moving all the glm implementation from the first (and second passes) over this project.

That's perhaps something I've not mentioned to you before, this project as a home engine... Well I've started and restarted it a few times.  The original was in DirectX9 way back; I wrote a page about it in 2004.

Then I re-wrote that in fixed function pipeline OpenGL; that didn't go very far, indeed I received a letter to cease and desist from an IP owner for that particular project's name was known in the wider world.  That IP has moved onto yet another owner now, so best of luck to them.

I however was knocked for six and it wasn't until 2018 that I started to look at a game engine at home again, spurred on by my change of job; and I really enjoyed that first pass, but it didn't focus on the graphics side of things, it was far more about the technology such as threading, memory management, and something you may have heard of, an Entity Component System.

All those pieces remain.

But this new project is very much just about my putting features together into a renderer, indeed the application level item is called "renderDemoApplication" and that is all this is.

What might the game be?  I'm still thinking about that, but I have three strong contenders and when I've picked one, you folks will be the first to hear about it.

Home Engine - Blender Export & Model Loading

Some progress in my home engine has been had, first of all multiple geometry rendering through my fixing up and better understanding both render passes and descriptor sets.

All of this is nuts and bolts, what I want to get to; just like the DirectX 11 rendered version; is to play with 3D geometry and placing objects into a scene.

To this end my hand written geometry is no longer hacking it, and you will have seen from my raw game engine posts I did use Milkshape 3D.

Milkshape, lovely as it is to my eye, is not a modern well featured 3D suite and sadly does not let me easily plug into the geometry live within it (one can only export and perform post-processing).

To that end I delved into how one might write a Blender add-on to export your own format...

Long Story short, here's my text intermediate output from just such a plug-in written in python.

The plug-in is far from robust, I followed the lines of another open source plug in and applied my own, copying it into blender with a batch file and clearing the pycache each time I perform an edit.

But it is doing the job I initially want, and I get my own format of file output.

And indeed I can load these into my engine...

I can work on the axis for the data output and start to ensure my camera system from the main client works as expected in vulkan; since the maths here is exactly the same code I don't anticipate any issues.

This should free my mind up to think about a lighting system and to finally get back to where I was with the coloured cubes in the DirectX 11 version and ultimately onto a full scene.

I have four constant things in the world, a ground plane, a skybox and a center marker flag (showing X, Y and Z axis respectively) with a quartet of boxes in the center of the scene.  This is the test scene for the "world" loaded into the scene graph.  And it's where I want this to be this time next week perhaps.

Since progress has been much quicker now I'm past the deep dive into learning the initial set up of vulkan.

I can also appreciate much more why folks just pick up an off the shelf engine, I myself make little prototypes in Unreal Engine 4.  But this kind of denudes me of the walk through these technologies.

For the most part creating an engine like mine, which is planned to have solid shaded low poly vehicular combat, well it's a solved problem, so I'm enjoying following in a lot of well trod foot steps to better my understanding of the processing, memory, mathematics and challenges in writing a game engine.

Home Engine - Set Course for Vulkan

Over the last year I've been experimenting with writing my own home game engine, initially I spent time writing my own line rendering and then a wrapper with SDL2 all using my own mathematics library as a refresher.

Then I set up an engine framework, switched my own none-peer reviewed and frankly shaky maths for glm and started to use DirectX11. You can see a clip in my earlier blog post here.

I dabbled with other graphics interfaces, but ultimately I wanted to land on one of the two big modern interfaces, either DirectX12 or Vulkan.... And since I'm a linux guy at heart Vulkan won out. The difficulty? Well, setting up a Vulkan renderer just to render the "Hello World" triangle is over a 1000 lines of code.... And well, I wanted to write a framework as I went so I could just port this Vulkan back-end work into my abstract renderer API for my existing DirectX11 driven engine. That is my next step, but tonight, after a few hours each night this week I've finally gotten to the 3D step and I'm so excited I wanted to share with you.

So here we see I have multiple pipelines & render passes. It's creating and submitting multiple command buffers, I have integrated ImGui very roughly too.

But this is the first time I've used the uniform buffers, descriptor sets and descriptor layouts to send a uniform buffer (MVP) to the shader and get actual 3D.

I'm strongly following this online tutorial.

Though I also have the Vulkan Programming Guide official book and I'm writing my own functions as I go so the porting to the existing Home Engine API is minimal (or rather as minimal as I can get it).

One of the key pieces of tech in the base engine (driven happily by DirectX 11 when I switch to it) is a full camera fly mode system and a scenegraph rendering models.  I've also got a python driven exporter written for my own model format from Blender (and desperately need to get better with using Blender).  And I have a Python driven project generator (which calls down into CMake quite neatly).

The engine is structured into six major modules:

Base - which contains a host of utility functions, data types and tooling assistance.

Ecs - yes, I'm writing my own Entity Component System.

Renderer - The abstract renderer API which the application level creates and uses in the main game loop.

Application - Main app & game loop, I'm using SDL2 as the main interface here for input & window management in a cross platform manner too.

Physics - A basic hand written rigid body physics implementation, at the moment it really only supports rays, cubes and sphere intersection tests and I'm thinking of shopping around for an off the shelf something.

Game - Contains specific game logic, like spawning, camera controls, input mapping and uses base to provide timing.

This is all very rough and ready, it's a project to keep me interested in my craft really.

And putting Vulkan into the mix has been a real nice way to resurrect where things where, as you may gather I had the DirectX 11 and basic camera system all working in October 2021; but I've been a busy boy since then, both with work, life and moving house....

Now the new office is done and I'm building models in real life, I figured it'd be a nice time to build them in virtual spaces too.

P.S. I have no idea why some blocks are in white backgrounds, very annoying of Blogger to do that?

I hate FizzBuzz and Will Analyse anyone who uses it

One FizzBuzz Interview so far.... News flash... The moment I am asked to write FizzBuzz during a technical test I intentionally stop and ask the questions from there on out.  I've done this a couple of times.  And you know what I learned... I learned a whole lot more about the interviewer and their processed than they found out about me.

Let me take you back to a technical interview I had about five years ago.

I drive up to the building, it is way out in the sticks; just about still in my home county, but so obscure it's in a part I have never ever been in.

I pull up and the building is like a box of black glass, no character, no soul, just a box.  The non-native tree species planted around it to discourage birds actually using the tree's and the austere grass puts me off.

Things don't get any better, for I walk through the door and there's no secretary, no desk, nothing... The lights aren't even on; this doesn't bode well.  I look around for a buzzer, a phone even a limp notice of what to do.  When a door opens and a man approaches offering me a hand to shake.  He leads me to a similarly deserted corridor with a chair in it and I'm left to my devices once again.

I read some inane notice about some award some member of the staff had received (this is important later in the interview) and I read the fire evacuation notice.  It's quiet, quiet and silent.  There's one door nearby with a few people within and another which is a closed office piled with stacks of desks..... Empty corridors - which would be dev spaces, massive rooms with a few people in them, no secretary and piles of desks?  What have I walked into?

About ten minutes later I'm ushered into this room, I've brought my CV a few piece of detail about me and some references to my technical projects at home; as well as references to some of the discussions I have on these very pages and my github.  You know actual technical things, which you can see me achieving.

I can't talk about my current work with them... Heck I can't talk to you guys about my work now.  But I do it, and I pay a mortgage; that's all the proof I can proffer.

Anyway, I seat myself and a lady explains their technical lead will come and perform the technical test, he enters a moment later with a massively overpowered gaming laptop.  I hate gaming laptops.

I much prefer a simple sleak laptop, a decent screen, nice keyboard and plenty of battery life; and I do my work on a remote machine via SSH or whatever machinations you wish.  I do not generally work on my laptop, it's a portal onto other more powerful machines which are tethered, and this way I can go take my laptop with me anywhere and get amazing performance from it, as I'm only interested in a bit of internet, RAM and then remoting into other services.

Anyway, he plonks this behemoth down and proceeds to tell me about test driven development... TDD.

I HATE TDD.

Now, I know test have a place, heck I really like doctest, including tests into code you've written, to ensure it is within specification is perfect, and validates your progress.

My hatred comes from writing tests up front, I find this backwards logic, and I don't think you could ever persuade me otherwise.  Define the scope of your input, validate it in your code (I often assert up front when data is unexpected or malformed, or at the very least bail early) and then you run your code, and then TEST the result.  I do not believe you should write a test and then further write the code to fulfill that test.

You will miss something, you will be thinking about the test as the solution, rather than the solution itself.

If anything writing the test and then producing the code to meet that test may very well be called TDD today, but when I was a lad it was called Prototyping.  And though I'm sure we can all agree prototyping is best done in a language different to your final product, it is not really prototyping when you are not producing the product, you're producing a test.

I find it wasteful of time, and often ignores enough scope, for instance I've seen a fair few hundred thousand up front tests written and then the code to meet them in my time, never have I seen a validation of the input data, the input has generally been hard set into the test.  That's not as wise a test-scope of ensuring the result after the fact.

(This does not mean test suits after the fact are the best, I just think they're the best for my style; not least as they are opt-in, for those times you are in a swing, and easy to add later).

So, TDD... Why is he telling me about TDD?

He does, he explains they have a method and that method is structured, he does not offer me any test framework specifically, instead he opens notepad and asked me to write FizzBuzz.

And the interview stops there; I don't think the guy realised it, but I stopped being interviewed at that point, I was not going to accept their job even if offered, but I wanted to see where this lad (and he was young) was going with his thinking.

This half empty hulking building and they're trying to hire engineers, and he is going on about TDD but FizzBuzz?

So I play total dumb, I simply start to write "int main"... And he surprises me.

"No No, start with the test".

Ohkakkkakay....

I have NO idea what test suites he has installed, I don't use G-Test or Boost-Test, I use doctest.  So I write a SCENARIO; he blinks and just takes the laptop off of me and starts to open some other window... I didn't know he would allow me to open some other window, this isn't my machine, I don't know what he has installed.

This sort of sums up TDD to me, the tools aren't easily defined.

Anyway, twenty minutes later and I just about have it Buzzing on 5's and I'm not very interested, I've come to a conclusion.

He isn't impressed; rightly so, but he has no idea he's just been investigated by an older and more cynical mind than his.

He leaves and no less than three people come into continue the conversation, and boy do they, they talk to me for a solid 120 minutes; I timed it... I was so bored.  But I engaged a little when they talked about their award winning technology.

I recall the poster for the lady who had won this award and mention her, by name.

"What, how do you, who has been talking to you?"

This is very strange, "no" I assure them "no-one, I saw the poster on the notice board".  One of the two girls flanking the chap in the middle here makes a note.

They talk to me about the company a bit more, they talk about their re-shuffling engineering and wanting to press their position forward to a virtual machine the customer hosts on hardware; previously they had been providing a dedicated box.

I'm terminally not interested in this boring black blocky building which now perfectly reflects the boring blocky people inside.

And I give up, I lean forward, elbows on the table, and I gather the copies of my CV from the hands of the three people and I say:

"Look, I've spoken with your technical lead, and I'm scared to my core of your processes, I don't like what I've heard, it seems you're struggling to recruit and there are reasons for this you will have to identify yourselves; I don't think I can help you"

They are scowling a little here, I'm sure they're far more used to candidates who want the job.

"Indeed, I think somewhere you have a beige room with ranks of screens and engineers in stuff suits not talking with a great board or wiki with all the failing builds traffic lit from your little test driven development board...."

At this point the lady on the left looks at the guy in the middle, who has a little switch to his eye.

"And such a stifling, control freak, desperate development environment is not one I wish to be part of"

They thank me, and I thank them sincerely.  And I leave.

As I come out of the building the sun has come around, the black glass face of the building is now not so black, instead I can see inside, there are three long benches of PC's all facing a pair of chaps at the front sporting bluetooth ear phones in one ear.  Above their heads is a large TV screen with a list of running continuous integration tasks and each has a traffic light.  As I watch one goes red, one of the two larger men touches the headset at his ear and an engineer on one of the benches glances up and starts to scrabble about on their machine.

It is a sweat shop.  I was not only right, but so happy I'm driving off.

I felt like going and knocking on the glass and cooing at them "Come, follow me, be free, be free".

But instead I just drove home.

Prior me: Just coming into edit this before it goes live on Friday, guess what I'm watching... 

And I disagree with the approach of analysing FizzBuzz; analyse the interviewer.

Embarrassing #3: Source Control

We spin far forwards not to around 2016; and the development manager wonders over to me and asks my opinion on source control software.  GIT.  Is my reply, just Git no other shenanigans.  But he wants a report.

I duly go take a look... I've suffered CVS, SVN and a myriad of other source control solutions, Git has always worked solidly for me, in over a decade I've had one fault with it - and was able to recover from that problem easily.  That is fabulous uptime for me and nearly zero support because anyone using my repo's could just learn (or google) a bit of git.

SVN was the other main source control solution I advocated, it's issues are many, but it worked for the problems I was dealing with (indeed on my youtube you will find subversion tutorials from back in the day).

This manager however wanted more than opinion, I wrote him a pro's and con's, I went to the public for their opinion and looked at a few closed source alternatives; like perforce.

Cost was a MAJOR issue for this company, the cost to install Git was zero.  The cost to install Perforce was nearly £27,800.  It was a clear win win for Git and vouched for it again, arguing it'd be better to pay for a whole other junior developer; or paying the ones we have now that little bit more each than going for Perforce.

(And I'm being harsh here, but I don't like perforce, I see the "game industry" using it as a default goto, but I do not see why, it has bitten me more than a dozen times in four years of heavy use - I compare that to once in over ten for git and it's just a no brainer even before the costs; and git will do the huge repo's and do them better than perforce can - which was the default argument before that git didn't do huge repo's or source tree's... erm, hello, Linux is completely on it and huge.... Microsoft have moved to it, so windows, edge and so much of their stack is all hosted on git repo's - stop arguing for perforce, it's embarrassing, seriously).

So this manager took away my report.

They bought Perforce, six months later, he's getting the nagware for license renewal, per seat.  £12,000.  This just beggars belief.

The manager was wrong, the money was spent, the company still disappeared with a whimper, it was a waste.  He should have been embarrassed by that; I doubt he was.

But the management should have been far more embarrassed letting him operate the way he did.