It's a C++ STL Rant : Constness and Vector Erase

I believe I am right and correct to be seethingly annoyed that this code does not compile...

#include <vector>

#include <iostream>

#include <algorithm>

struct Test

{

    const float value;

    int changable;

};

int main()

{

    std::vector<Test> items;

    items.push_back({42.5f, 0});

    items.push_back({1.23f, 1});

    items.erase(items.begin());

}

It just really annoys me this won't compile, do you know why?  Don't cheat, just have a look why does this PERFECTLY REASONABLE AND ACTUALLY I THINK CORRECT CODE NOT COMPILE?

Yes, because of the ISO Standard, and it's really annoying.

ISO/IEC 14882:2003 23.1 [lib.container.requirements] / 3:

The type of objects stored in these components must meet the requirements of CopyConstructible types (20.1.3), and the additional requirements of Assignable types.

The compiler could have been told this far more simply if the std::vector had an earlier and cleaner static_assert that the type is copy constructible before it got into the weeds of the real message we get.

For can't compile because it seems the standard implementation of erase results in a bunch of move semantics in the vector, which then devolve into copies.  And to my simple monkey brain it's all just babbling horrible template junk that comes out the compiler as well.

You can of course move a struct of type Test perfectly well, or at least mark it for move (cast it thus):

#include <vector>

#include <iostream>

#include <algorithm>

struct Test

{

    const float value;

    int changable;

};

int main()

{

    std::vector<Test> items;

    items.push_back({42.5f, 0});

    items.push_back({1.23f, 1});

    Test bar { 123.456f, 2};

    items.push_back(std::move(bar));    

}

This compiles and runs just fine!

But it's not copy assignable.  Let us take a look at the compiler spew:

In file included from <source>:1:

In file included from /opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/vector:60:

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:427:4: error: static assertion failed due to requirement 'is_move_assignable<Test>::value': type must be assignable

          static_assert( __assignable::value, "type must be assignable" );

          ^              ~~~~~~~~~~~~~~~~~~~

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:495:22: note: in instantiation of function template specialization 'std::__copy_move<true, true, std::random_access_iterator_tag>::__copy_m<Test>' requested here

                              _Category>::__copy_m(__first, __last, __result);

                                          ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:522:19: note: in instantiation of function template specialization 'std::__copy_move_a2<true, Test *, Test *>' requested here

    { return std::__copy_move_a2<_IsMove>(__first, __last, __result); }

                  ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:530:8: note: in instantiation of function template specialization 'std::__copy_move_a1<true, Test *, Test *>' requested here

                std::__copy_move_a1<_IsMove>(std::__niter_base(__first),

                     ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:652:19: note: in instantiation of function template specialization 'std::__copy_move_a<true, __gnu_cxx::__normal_iterator<Test *, std::vector<Test>>, __gnu_cxx::__normal_iterator<Test *, std::vector<Test>>>' requested here

      return std::__copy_move_a<true>(std::__miter_base(__first),

                  ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/vector.tcc:179:2: note: in instantiation of function template specialization 'std::move<__gnu_cxx::__normal_iterator<Test *, std::vector<Test>>, __gnu_cxx::__normal_iterator<Test *, std::vector<Test>>>' requested here

        _GLIBCXX_MOVE3(__position + 1, end(), __position);

        ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_algobase.h:656:44: note: expanded from macro '_GLIBCXX_MOVE3'

#define _GLIBCXX_MOVE3(_Tp, _Up, _Vp) std::move(_Tp, _Up, _Vp)

                                           ^

/opt/compiler-explorer/gcc-12.2.0/lib/gcc/x86_64-linux-gnu/12.2.0/../../../../include/c++/12.2.0/bits/stl_vector.h:1530:16: note: in instantiation of member function 'std::vector<Test>::_M_erase' requested here

      { return _M_erase(begin() + (__position - cbegin())); }

               ^

<source>:20:11: note: in instantiation of member function 'std::vector<Test>::erase' requested here

    items.erase(items.begin());

          ^

1 error generated.

Compiler returned: 1

We read this from the bottom up, first erase needs to call "_M_erase(begin() + (__position - cbegin())) what on earth?  Well it appears this is trying to perform a copy-move of the rest of the vector; the rest of the stack all the way to the top shows us various iterator types to the top two messages where we have __copy_m and then a compile-time static assert that the type must be assignable.

With the intentionally const member we can't compile we therefore have two options, first we make the member non-constant.  This isn't very practical because the type is derived from a bunch of runtime values, they're not known upfront and can be different for each instance of the struct.

The second option therefore is even less appealing and looks like this hideous mess:

#include <vector>

#include <iostream>

#include <algorithm>

struct Test

{

    const float value;

    int changable;

    Test& operator=(Test pOther)

    {

        float& mutableV { const_cast<float&>(this->value) };

        mutableV = pOther.value;

        this->changable = pOther.changable;

        return *this;

    }

};

int main()

{

    std::vector<Test> items;

    items.push_back({42.5f, 0});

    items.push_back({1.23f, 1});

    Test bar { 123.456f, 2};

    items.push_back(std::move(bar));

    items.erase(items.begin());

}

We define this assignment operator performing this const reference magic and just.... ergh, don't do this folks, just don't.  Not only is this an excellent lesson in const member flagging in STL and it's pitfalls.  But then it's also a lesson that const doesn't really mean const.  AND it's a lesson on whether you want to keep a constness or nor given such awful hacks to get around it.

I could even go further than this, the operator= could be made private and the STL type "std::vector<Test>" could be made a friend, or the __copy_m could be made a friend; though in that last case the MSVC implementation of the STL will be different, so we'll have to work around platform differences too.

No, the most annoying thing here is picking open quite why __copy_m wants to end up assigning.  I am still digging into why, but oh boy am I annoyed by this one.  For in a module I unironically had to repeat at University solved this in 1996.... I would use a linked list, and simply pluck out the value... or I would skip to the index of the item to remove and copy those after (if they were host memory contiguous then memcpy speed achieved) and just write over and this erase is achieved.

To just see this wanting to copy assign at the element level is just hideous, and I hate it, and I with the ISO standard would change in this regard,

Story Time - Fantastic Rack Mount Mistakes - Part 33 and 1/3

It has been a very long time since my last story time, and so this one comes to you with an unknown number, I am however well beyond any possible NDA or secrets for this one, so I can finally share (it's been 20 years yesterday).

I had a job in a data center, nothing too special, but I loved it.  The loud room, the air conditioning, the rack upon rack of machine, I absolutely loved it.  We're talking 2002 here, these were the 2U high style of machine I use a lot today, the odd 4U or 8U JBOD box too, loved them all.

I had all my wiring terminated and was beautifully dressed, I wasn't in charge of the whole place, just one isle, but I took my time whenever there was an issue to run new cable bundles beautifully dressed along the cabinet runners into each cabinet top, seven cabinets to a row, two face to face on the cold row and then the back of these into the hot, I had a hot row run by a lovely guy called Jerod next to me, he was French I think, lovely guy (shout out to Jerod if you're reading!).

The other side I had the work bench, a coveted space, between the hot and cold isles were thick plastic strip drapes, like you'd get in a meat packing plant.

We would patrol our isle, identifying any issues or disks showing physically bad, schedule them in for late night fix ups and then to into the communal office where we each hot desked.

Each of us had a laptop and would sit and connect into the monitoring suite remotely, with a big screen we connected via VGA to our laptops and dozens of spare keyboards we could quite often spend the night shifts doing nothing but playing games; Medieval Total War was the order of the day for me at the time; some of you know of my origins in the "game industry" take me back to hacking about and modifying MTW (Sorry Tom, whom I now work with, for fishing about breaking your baby).

Anyway, about three months into this job another tech on the day shift came to me about Isle 4, the far end of the room, he was getting drop outs at intervals during the day, but none at night, he was trying to figure out what might be causing it.

We took a look at his logs and sure enough on his disks he had 250-700ms delays for seek times on his platters, he also had a higher than average fault rate on disks, apparently he was also rebuilding raid arrays a lot.

He asked us in the night team to watch over the kit live; which we duely did.... Quiet as a mouse, nothing not an issue ever in about three months of monitoring.

Meanwhile in day light he was tearing his hair out; I've got to admit to genuinely not remembering his name, but he had long greasy black hair and was from Northampton which he told me more than his name... "Hello I'm <forgetable>, I live in Northampton".

But forgettable as he was I can't forget the day he handed that isle over to me.

Stumped as to what was going on, and frankly pulling rank, he shuffled Jerod down to Isle 3, himself into Isle 1 (my isle) and all new machines were coming into isle 2, which mr senior with disk failures was going to be setting up.

This really irked me, my isle was by far the best presented, when sales wanted to show new accounts around they showed them my isle and they all laughed over the name of the cabinets; we couldn't name the machines, most of them belongs to customers, but we could sticker and name the cabinets themselves, so they all got Bond Theme names.... Jaws, Octopussy, Moneypenny, Q and M all had simple servers, processors more or less and my two JBOD cabinets were Sean and Roger.  Stencilled on in lovely white enamel paint.

So I was hoofed off to isle 4, dark, at the back, no work bench... and immediately set about making it my own, I redressed the cable bundles (which annoyed Mr Senior, though he loved my already done ones, my dressing his was seen as an overt declaration of war).  It's so simple too, cable tie around five, cable tie through the middle to break them into three and two sets then pull.... Today you can get cable combs and I find them so sexy.

Anyway, dressed to impress what was not impressing was the fault rate, and instead of it being the accepted norm; as it had been for Mr Senior for months; it was now a big deal because I was in charge of the isle.

I set about trying to work it out, I could not figure it out.

At night, when I was there it was all fine, the day guys saw nothing; they pretty much just tended the machines remotely, no maintenance happened between 7am and 7pm so it was up to the night guys like me to swap disks, rebuild arrays, fix cables and install new machines.

So after two weeks with the log I got showing all these faults, like what was causing a disk which had worked all night and most of the day suddenly to show 500+ milliseconds seek!  It made no sense.

Anyway, 2002 was coming to a close, December 21st rolled around and I offered to cover a couple of the guys shifts, they had kids it was nearly Christmas and we worked Christmas Eve and half Christmas Day, one guy on on guy off for 6 hours at a time those two days, I took two shifts back to back.

Alone in the office, I watched the monitors and played games.

No spikes.... What gives, no spikes on the DAY I am actually here.

Hold on, I'm here, but no-one else is.

Could it be human interference?  I checked electrical circuits for noise, I checked lights, I checked if the air conditioning was affected outside and in.  I checked everything, no signs, no peaks, no noise.

HANG ON.  NO NOISE!

Spinning disks can be affected by loud noises, specifically by vibrations, I'd seen this demonstrated when doing my Compaq certification training.

There was no noise, could it just be vibration?

We were on the first floor (for anyone in the US, this is the floor above the one on the ground) so we are one storey up.  There was the main reception below center of the room, to the right where isle 1 is would be a hall way void with offices leading off it, under isle 4 would be a toilet and a changing room.

I went down, put the shower on, came back up.... Nope.

Then I took my laptop with me, no wifi, but I could plug it into various office ethernet ports around the place as no-one was in.

There I am Christmas Day, banging doors, flicking lights, flushing loos and then pouncing on my laptop to see if it affected any of the disk activity I was artificially running up.

Nothing.

Defeated I logged my time, handed over to Jerod and went to have my Christmas dinner.  I had 12 hours to think of something for Boxing day.

I walked into Mr Senior asking why there was CCTV of me "Dashing about with my laptop in random offices".

I just said I was trying to check light circuits for his disk issues; he made it abundantly clear they were my disk issues and left me to it.

Boxing day, the sales had started, the building was at the corner of a large commercial estate, there being a newly built Ikea across the road as the crowds rolled in and their stock levels fell they would be due a delivery soon.

I still poked around the office and the eureka moment came on the 27th December 2002.  For a large lorry was rumbling past the office, I had to wait for it to pass, a massive blue IKEA clad lorry.  I went into the office, logged into the monitoring and sure enough there was a trace of a large disk issue.  Times 4 minutes before; when that lorry went past!

I didn't hear it, I didn't feel it... But had the disks?

I waited and watched when a few lorries passed during the fairly quiet week between Christmas and New Years, nearly every heavy lorry going back resulted in some affect on the disks, vibration was being carried into the ground and I guess up through the building.  I will be honest, I could not tell.  But every spike I saw was timed with a lorry.  In fact I soon let Jerod in on my idea and plan to fix it and so he watched and I monitored and when he came in I could tell him when a lorry has passed!

I was not about to shout about this to Mr Senior, instead I set about fixing it.

I ordered a mat of 1 inch thick rubber, the stuff you mount washing machines on in your kitchen.  I already knew the racks pretty well, they were bolted at each foot, I'd need a torque wrench to unbolt them and I could use two of the hydraulic scissor platform lift trolleys we used to move machines about to lift the rack ever so slightly.

I didn't want to do this alone, so Jerod was roped in with the promise of a take away pizza from the glorious; but long gone; parlour we loved.

Mr Senior handed over to me and Jerod that night, it was not uncommon for two of us to be on at night, especially if there was work to do fitting something out.  It was normal to have two people when we were lifting machines too.  But Mr Senior would have been apoplectic if he'd know what us two kids were about to do.

I unscrewed the first rack from the floor, swept the dust out and used a wooden baton to bridge the lip to the metal of the scissor lift and I cranked it.... The cabinet moved, I nearly wet myself as it looked like it was going to topple, then Jerod jacked his side up on the other side and it came level, with an inch to spare I slipped in two of the pads.   He slipped in his two and at a shout over the loud AC we lowered.

I then used a phillips head screw driver to puncture the rubber and a knife to dig a bit out through each bolt hold and fastened the bolt back through, not too tight, just tight enough.

It was sweaty work in the hot isle as I was, but we did the first three of seven that night.

I went home and decided to pre cut the squares and use a drill to cut the middle out of them,the next night all seven were done.

Our disk fault rate when to zero.

That was my last time as an IT minion; I went back to programming soon after, and Mr Senior never was told what we did... His training should have told him.

What did Don do?

First things first, we're going back to the latter half of the 90's here; and I've mentioned Don before in one of my older posts, skidding into hero mode.

Don was a programmer, he was kept in a little plexiglass bubble of his own and spent all day in what looked like vi (certainly a terminal based text environment) writing what I think was C for the NCR minicomputer running Sco Unix I mentioned before.

Whether this is actually true, I don't really know, for he spent a lot of time sitting with his hands crossed over his head waiting; at the time I didn't know what he was waiting for, at home I had a Pentium 100Mhz rocking 16mb of RAM and running Borland Turbo C++ for DOS I had a lovely little progress window showing me it was compiling; I had not met the glory of GNU yet.

So Don would stare at a massive (like 19 inch) monitor, he had a Wyse VT100 next to him too, and he would not stare at that very often.  He would also make copious notes on paper and seemed to program very deliberately, looking and reading through the information at hand.

He did this, almost without interacting with anyone else for the whole year I worked with him; looking back on it I wished I had been more proactive and just asked him more about it, as he was clearly programming C around a Unix system, which is a) uber cool and b) uber cool.

It always struck me quite how long he thought about and maybe designed anything before acting.  You could say this was because he was mulling and considering the problem, he was being a good engineer, with forethought and wanted to get things right?

No, see I think he was really just in fear of how long the compile would take...

Just like me with the project I'm working on twenty five years later.

Confessions of a Linux User

I've never understood this one, but I've felt bad for it for a while now, so I'm going to come clean.... At my last employer they had a very strict network/internet traffic policy, like everything was locked down.  I had a few issues with this because using Windows and Chrome it didn't just block the at the DNS or URL returned level, but it actively broke open HTTPS packets and inspected their contents.

YEAH I KNOW RIGHT?

This was meant to be secure, but it clearly wasn't and I asked about what was opening these packets and inspecting the contents and bless 'em the IT department said it's not... It clearly was.

Because, slip over to a none-Microsoft operating system, even one running in a VM on the self same windows host and it could get out to the internet and retrieve whatever it liked.  So it wasn't just blocking the URL it was inspecting the actual traffic, the actual supposedly secure traffic wasn't secure.

I'm pretty sure this would have proven illegal; just like Scunthorpe Council was found reading its employee's email.

Either way it wasn't that they were doing this that actually bothered me, it was that when asked they said they weren't.  Which meant one of two things, either a) they were and were lying or b) something else was and they didn't know.  I believe it was a case of option A.

For you see besides watching copious amounts of YouTube and listening to even more music, which no-one else could I saw the darker side of this power I had through Linux.

You see the packets were being broken over to inspect their contents because IT wanted to filter certain search terms or words, you know like porn or weapons, nasties.  HTTPS connections can't be inspected (at least between your browser and the receiving/signed server they shouldn't be) so the software on the Microsoft pipe, or Windows itself, was compromised in my point of view.

That's not to say I was looking for tits, but I could have if I wanted to.... Tux!