Tech Tribulations #1 : Smartcard Release Drama

It has been a very long time since a story time, so I thought I'd go over one about a software system I wrote from the ground up to secure the service to a machine; so I worked for a company which sold a whole machine to the customer (or leased them) while ever the buyer had the machines they would run.

In late 2014 the higher management realized this was an untapped revenue stream, and much to the annoyance of the customers, it was decided that a system update would go out; which the customer had to take to get any new content; and in this update they would have to also have a smart card reader installed and a card inserted which would count down time until it ran out.

Metering essentially, but "metering" had a whole other meaning for this system already, so it was just called the "Smartcard" system.

Really it was a subsystem, bolted into the main runtime as a thread check, which would wake at intervals, query if there was a card reader on the USB at all, check it was the exact brand of card reader (because we wanted to limit the customer just being able to put any in, they had to buy our pack).

And then it would query the card and deduce credit/count down if we were beyond a day.

We tried a bunch of time spans, hours, minutes etc, but deducting was decided to be after we accumulated 24 hours of on time, every 5 minutes an encrypted file on the disk would be marked, after 24 hours worth of accumulations the deduct would happen.

We tested this for months, absolutely months and to be honest we thought it was really robust.

Until it actually went into the customers hands, we suddenly had a slew of calls and returns, folks unhappy that they were inserting the card, "testing their machines" and suddenly all the credit was gone, and they were asking for a new card all the time.

At first we could simply not explain this anomaly, we had the written information about the service calls, replicated what the folks were saying, it all checked out fine, we got our increments, we could inspect the encrypted file and see we were accumulating normally and deducting normally.

I worked on this for days on end, we had to test for real, we did all sorts of things, power drop tests, pulling the card tests, all sorts.

The machine checked out, end of.

What had we missed?  What are the customers doing?  Testing the machine, okay, what are they testing?  The content, how does the content work for this update?  Well, seems that the customers didn't trust their testers or engineers, so what was happening was instead of testing for real they were doing what we called "Open door testing".

You see, when you close the door you accumulate and deduct, the machine is in operation normally as any user their end would have it operate....

Door open mode however, was intended to be used by service engineers, when the machine was deployed; so it is still in operation, the machine is in the field, but the door is briefly open to check things.

But these customers didn't trust their engineers in their warehouse, so they were not giving them credit to check the machine properly, they therefore tested in open mode... for days....

They accumulated massive operation debt with the machines in door open mode for days.

The moment they turned them off, happy they were working, and shipped them to sites they'd arrive on side immediately be turned on finally after so long in proper door closed operation and they'd instantly deduct the massive debt the warehouse team has accrued.

This was intentional.... But their use of the door open mode was an abuse, and one we had not even thought about.  We didn't even clock how long a machine sat in door open or door closed mode, worse still when in door open mode and test things on the machine ran at an accelerated update rate, we ticked over 10x faster to allow faster testing... The result was in just 3 days of warehouse door open mode testing they could accrue 30 days of operational debt.

That was a fault, one I could tackle with the team.  But changing the user habit of leaving the door open was harder...

We had to work with the user, and their patterns, we suspended the system for a short while and issued a new update, but the first customer taste of this "pay as you go" approach was a sour one.

Then things got bad....

Yes, you might think they were already bad, but they got worse.

A month later, all the above was resolved, and we thought things were settling down... Until we suddenly all hell broke loose.

EVERY MACHINE WAS LOCKED.

There were dozens of reports of their just not working, they had done their daily reboot and all of them reported a security fail on the smartcard....

All hands on deck, are our machines in the test pool doing the same?  Nope.

Is there something special going on?  Have clocks changed, is it a leap year, has the sky fallen on Chicken Little?

We honestly had no idea, there was no repeat in any of our test pool, no repeat on our personal engineering rigs, there was essentially no reason for this failure.

The only answer in such a situation is to observe or return one of the machines exhibiting the problem.

A lorry was sent and a machine brought back, under the explicit instruction not to open it nor change it, and the customer was not to keep their smartcard (it was theirs, but we would credit them a whole new card for the inconvenience).

Several hours spent staring at the code, and running checks by lowering cards so they would expire, or pulling the reader out and inserting it again we had no answer.

Before that arrives back with us however lets just think about the "smartcards" we used in our daily lives; our bank cards, they go into a machine we enter our pin and we remove them again.  Then how about cards like your gas meter, or you go see your GP, they have a card you insert into a machine and it stays there all the time to validate they are the GP, or they keep your meter in operation, if you have Sky TV and a viewing card; same thing, it is always in the device.

These machines are the latter kind.... Those cards are rated to have power on to them for long periods of time, as a consequence they cost more money than a card you only insert transiently...

And this company I worked for had very canny buyers, too canny.  Because they spotted a smartcard which used the same protocol... but was significantly less money to buy!

The difference?  You guessed it, it was the transient use variant.

The broken machine arrived, we powered it on, fail.  We open the door, remove the smartcard and sure enough on the rear of the plastic behind the chip the plastic is brown, burned.

The card can not be electrically trusted!

We highlight this and send it back to the buying department, they fouled up, they changed the hardware after we certified it, essentially sending an uncertified machine out.

A huge issue ensued about this, as this wasn't well understood that we had been provided and advised one card type into the update set, but of course the buyers would not accept it wasn't the same until we literally had the specifications of the card side by side we could see a digit difference in the part number and looked up the datasheet where clearly it said that the transient card was only rated to remain in a machine for 10 minutes.  More than enough for an ATM.  But a "security gating" card, as we wanted, they are rated to be inserted continually for 36 months.

Debugging Techniques

 As a programmer there are many approaches to a take in solving a problem, I remember in my A-Level they talked about trace tables... A lot... I've used them now and then, but almost immediately machines were running so quickly as to make humanly computing any meaningful number of permutations near impossible... and we're talking only a 25Mhz machine.

Far more useful therefore was a decent watch value in a debugger, improved immeasurably by conditional break statements.

Even so there are always times when you need that holistic overview, my favourite approach for this is a huge wide whiteboard.  Many an hour have I spent around a whiteboard laying out the schemes of madmen and managers; the greatest compliment is then when finding that said folks have taken a cheeky photo and your whiteboard machinations appear in the manual.

But one of the best problem solving solutions is to share it, a problem shared can often become a problem halved; unfortunately not always for a programmer.  You see people are a creature of opinion and social status, this means you can often run into folks who believes their opinion is more valid than yours, which maybe so, but if their opinion means starting the whole problem over again it can often be more detrimental than asking for their help.  For help becoming opinion leads only to their being demons in the cupboard.

So what then?  Talk to one self?

And you know I think really rather that is one of the key approaches, talk to yourself.  Quite useful in the current pandemic too.

We have a term for this in Computer Circles, and we detract from the self by making our conversation focus on another object, such as a rubber duck... https://en.wikipedia.org/wiki/Rubber_duck_debugging

Or of course you can find your own devices....

Programming : Python MySQL Connector Debug

Today, I was asked to look at a server for a friend, their problem... "It just stops working after a few days"... A few days turned into "between three and five".  Doing some mathematics I found they had between 125 and 350 unique visits to the server, each unique visit represents one customer or one remote unit of their fleet.

They relay their data from these to individual database instances on one MySQL Server, so there is about 30 customers each with many unique databases.

The problem?... Well, I find this very distressing, as they open one connection for each arriving remote client, use it and then they closed it... Right... RIGHT?!??!!

import mysql.connector

l_total = 0
while (True):
    # Count
    l_total += 1
    l_res = l_total % 100
    if l_res == 0:
        print (l_total)

    # Open a connection
    con = mysql.connector.connect(user='root', password='***', host='localhost', database='Pickles')
    cursor = con.cursor()

    # Query
    query = ("SELECT * FROM VeggiePatch")
    cursor.execute(query)

    # Retrieve the data
    cursor.fetchall()

    # Close the query cursor
    cursor.close()

    # Close the Connection
    con.close()

This is my test code based on the way their production code works, as having read the error log I see the problem is in the connector constructor and delves down into the networking code.

This of course crashes after around 33,000 cycles.

They're not willing to change their script "willy-nilly", I in fact think they're petrified I've found this problem.  Googling around I don't find any official explanation of this error, only anecdotal forum posts about the MySQL Connector not cleaning up after itself and so reusing the sockets fails over time.

The better solution is to garbage collect the connection each cycle...

import mysql.connector
import gc

l_total = 0
while (True):
    # Count
    l_total += 1
    l_res = l_total % 100
    if l_res == 0:
        print (l_total)

    # Open a connection
    con = mysql.connector.connect(user='root', password='***', host='localhost', database='Pickles')
    cursor = con.cursor()

    # Query
    query = ("SELECT * FROM Tickets")
    cursor.execute(query)

    # Retrieve the data
    cursor.fetchall()

    # Close the query cursor
    cursor.close()

    # Close the Connection
    con.close()
    con = None

    gc.collect()

I also tried to garbage collect each time I printed the the "total", each 100 passes, but this still crashed, the fixed loop here has so far done just under half a million cycles without issue....

Typhoon Fighters, Old and New

I was doing some research about the Hawker Typhoon Fighter Bomber in World War 2 last night, this was to help feedback against a moderated bug report on the Gaijin forums for War Thunder.

As part of my research I spent a good two hours after that dismal England match against Uruguay reading and checking figures in several books, it took so long because one of the references I wanted to pull out of my archive was from the Imperial War Museum's "Images of War" publication, and I could not find the right edition (there being 52 of them).

Anyway, I did find it in the end, and after checking with Janes, I hit the internet to re-evaluate the original digital source links I'd passed over in the post.

They all married up, roughly, and my observations of the climb performance in game were much higher, so there's clearly something wrong with the flight model.  I don't know if its a bug in how the sustained climb acts, or just a bug in the drop off of engine performance at height... But whatever it is, its not firmly in the hands of their moderators.

I like that they're handling bugs like this in a moderated fashion, it stops the dross of trolling and random posting which was the bane of my life when at the Lordz, trying to figure out what was an actual bug report and what was just people having a go.

I was however interested to see this article from the Telegraph come up, when I was searching online about the climb rate of the Typhoon.

In case you're not aware the Eurofighter as it entered RAF service was christened the Typhoon, unlike its name sake this new generation jet fighter can climb, oh boy can it, I've seen them in the flesh, a diamond formation of four going vertical over the Lincolnshire countryside.

Maths: Zero Factorial in Code

Sometimes maths really confuses me... I have to be honest, I work with computers because I'm pretty good at asking them for results for things, and I know really quite well how to feed a calculation into the machine.

But, the source of these calculations is generally going to be mathematical and I find maths just to be so annoying...

Here's the information from Numberphile about Factorials...

Right, so they started with the factorial of 5 (5!) being 4 x 3 x 2 x 1 = 120.  And yes you can do this and ask your computer code to do it for you:

int factorial = 5 x 4 x 3 x 2 x 1;

cout << factorial;

Not problem, we can even write a program to do this function for us, quickly...

int factorial (const int& p_Factorial)

{

int l_result = p_Factorial;

int l_temp = p_Factorial - 1;

while ( l_temp > 0 )

{

l_result = l_result * l_temp;

l_temp--;

}

return l_result;

}

This code follows the first example they give in the video, and it presents correct results for all factorials of 1 and above.  This also matches with the graph drawn in the video.

This also goes back to what is stated in the video around the 0:45 second mark... "you can go down to 3 x 2 x 1", does Dr Grime mean you can take factorials down to one?  If so, then this code is correct, and this rule controls the while loop in there.

Except of course zero raises its ugly head, and breaks everything, now to explain zero the pattern above is not followed, a new pattern is invented, and this is where it gets sticky for computers and my brain, because the new pattern is not multiplication, its division...

If we approach the pattern above as multiplication, then my brain tells me the pattern, the parameter even, for the function starts 0, and then ends.  So to my mind zero factorial is immediately zero and the function ends:

int factorial (const int& p_Factorial)

{

int l_result = p_Factorial;

if ( p_Factorial > 0 )

{

int l_temp = p_Factorial - 1;

while ( l_temp > 0 )

{

l_result = l_result * l_temp;

l_temp--;

}

}

return l_result;

}

Or in fact I'd like to make the function throw an "invalid argument" and only let it calculate for integers of value one and above.

But this is not the pattern held to, no as you can see the pattern becomes "Take a factorial you've not calculated and use it as a parameter into the local calculation" this is a bit alien to a programming language function, take something you've not calculated as the parameter... what?

So, the function becomes something cyclic, or even two functions:

int factorialPreStep (const int& p_Factorial)

{

int l_result = p_Factorial;

if ( p_Factorial > 0 )

{

int l_temp = p_Factorial - 1;

while (l_temp > 0)

{

l_result = l_result * l_temp;

l_temp--;

}

}

return l_result;

}

int factorial (const int& p_Factorial)

{

int l_result = factorialPreStep(p_Factorial+1);

l_result = l_result / (p_Factorial + 1);

return l_result;

}

Now, the function "factorial" will always need to be passed 0 or higher, and it'll therefore always pass 1 or higher to the pre-calculation.

The full application therefore is:

#include <iostream>

using namespace std;

int factorialPreStep (const int& p_Factorial)

{

int l_result = p_Factorial;

if ( p_Factorial > 0 )

{

int l_temp = p_Factorial - 1;

do

{

l_result = l_result * l_temp;

l_temp--;

}

while ( l_temp > 0 );

}

return l_result;

}

int factorial (const int& p_Factorial)

{

int l_result = factorialPreStep(p_Factorial+1);

l_result = l_result / (p_Factorial+1);

return l_result;

}

int main (int p_argc, char** p_argv)

{

    cout << factorial(5) << endl;

    cout << factorial(4) << endl;

    cout << factorial(3) << endl;

    cout << factorial(2) << endl;

    cout << factorial(1) << endl;

    return 0;

}

And we see the output:

Right, so this code works?  With zero?...... HMmmm, lets see....

Yes, it works for zero factorial, which as per the video comes out as one, but this was by using the division pattern... 1 / 1 = 1, that's fair I get that... But it is not the same as the original multiplication pattern.

The original pattern, the original function, is a x b x c... so 5 x 4 x 3 x 2 x 1.... What is that pattern for one factorial?... Well the pattern is just 1... so what's the pattern for 0... well it's bleeding 0....

As a programmer this is an obvious avenue of research, we like to find multiple ways to get the same result, usually so we can find the most efficient solution for runtime, but what we don't like is two seemingly matching patterns, giving different results...

So for this code we could have written:

    cout << factorial(5) << endl;

    cout << factorial(4) << endl;

    cout << factorial(3) << endl;

    cout << factorial(2) << endl;

    cout << factorial(1) << endl;

as:

    cout << (5 * 4 * 3 * 2 * 1) << endl;

    cout << (4 * 3 * 2 * 1) << endl;

    cout << (3 * 2 * 1) << endl;

    cout << (2 * 1) << endl;

    cout << (1) << endl;

    

The pattern is right, the results are right, the code is right, but then we add 0!...

    cout << (0) << endl;

    

And it most certainly is not right, the pattern is broken, so was it a valid pattern?  Well it must have been because we use this pattern in our loop to calculate the prefactorial step...

Argh, see my head just exploded.... I hate maths for this.

And it only hurts my head more as I understand my solution code, and that solution code works... GRRRR.

Code::Blocks - A few Tips

I'm trying to set up a new Linux box to do some code (C++) in code blocks, now I do like how easy linking with multiple libraries is with code blocks - rather than fiddling with the command line or creating a make file which can obfuscate your build results...

But I'm not a huge fan of Code::Blocks, my biggest problems at the moment is I have a huge screen resolution, but no-matter how big it never feels like I've got enough workspace area available, it sort of feels hemmed in, this mainly comes from the poor way you can't layout editor windows side by side, or horizontally.  Forcing you to feel one file open at a time.

Then add to this that some of the editor settings pages - like "Settings->Environment" - full the whole height of the screen.  So you're there trying to edit the editor and you have this whole page filled with settings, and just as annoyingly you can't click off the editor settings back into the editor to look at something you've changed... You have to accept the settings, close the settings page and then you can see what you've done, this disjoints what you've changed from what you're viewing.

But I have a couple of tips to help you make things better.. First of all, pop into the View table and turn on the TODO list

Dock the panel for to-do items wherever you like...

And then you can add simple "// TODO MESSAGE" type items into your code to remind you what you're up to.

Another tip, if you want to maximise your coding area, you can close the messages windows - where the build output and messages go - but then getting them back open is a pain... To open them again, just press F2!

And finally, another one to maximise your working space, the editor tab showing the filename you're typing in.. Double click it... This will move into the maximised area version of the editor (by default) its a new theme called "Code::Blocks::Minimal".  If this is not setup then you can go into "Settings -> Environment" and then select the "View" (eyeball) on the right, and you're looking to make sure that this setting is selected:

Corrupting my Heap

I've had a hell of a few days, the family thing has droned on, but I'm fast washing my hands of the situation - they can go masturbate in their own mess as far as I'm concerned, this makes the wife and I the bad guys, but logic will one day prevail, most likely when my mother can't interrupt people and pontificate utter bollocks in her rude manner.

Anyway, what about this Heap Corruption?  Well, what a debugging session I've just had, it lasted nearly 3 hours - that's a long one for me - especially with my C++ code.  That's not to be big headed, but I have evolved my coding style and used the libraries (boost & std) in the manner I do without issues all the time, so to suddenly find a bug - one corrupting the heap - was a bit of a nasty sting.

I can't go into specifics, due to the nature of the work I was carrying out for my employer, but here's the gist...

A function was bound with boost::bind... And was called back from another location in the code at regular & mutually exclusive intervals, so on a timer... Here's pseudo code:

#include <memory>

#include "TimerFoo.h"

using namespace std;

void baa (const bool& p_Error, const bool& p_Cancelled)

{

   if ( ! p_Error && !p_Cancelled )

   {

      // DO SOMETHING

   }

}

unique_ptr<Timer> l_MyTimer = unique_ptr<Timer>(new Timer(boost::bind(&baa, this, _1, _2)));

l_MyTimer->Start(5000);

Right, and after waiting the 5 seconds the baa function would be called, and suddenly my application would crash horribly, inside the boost libraries.  Either in the function pointer headers, or even in the unique pointer header of the std.  It was an utter mess.

And the call stack gave no clues about the problem, it neither helped workout quite what had run.

The problem turned out to be a problem inside the "DO SOMETHING", which was calling into the psapi and trying to use the TCHAR pointer somewhere incorrectly.  I removed that code and its all fine again - but so scary a crash.

It has spurred me to want to update my personal coding standards and include the rule "If you have a boost::bind bound function, and suddenly your code crashes, rem out that function's body and re-run".  This is essentially how I found the above bug, I removed the whole guts of the callback and voila the code ran, but with the heap corruption it was hard, nigh impossible, to see what had actually gone wrong.

C++ (Sprintf in String) STD Bug (cout)

I have noted a curious difference, perhaps one could say a bug, in the Visual C++ runtime when using std::string with cout...

Now, first of all, what I'm doing it not standard fiar, but lets jump to some code and see what we think it should do...

#include <string>

#include <iostream>

using namespace std;

int main (int p_argc, char* p_argv[])

{

string l_output;

l_output.resize(100);

for (int i = 0; i < 1000; ++i)

{

memset(&l_output[0], 0, l_output.length());

sprintf(&l_output[0], "\r%i\0", i);

cout << l_output;

}

return 0;

}

I expect this code to run in a loop, to output on the same line the numbers 0 through 999, without the newline character being involved, as you can see my sprintf call just returns the carriage, outputs the number and nulls the string for good measure.

But, this is not what happens the call to "cout" results in newlines each call... it seems passing the std::string directly to "cout" causes this, it treats the string as a whole line.

We can stop this with a code change to force the type of char* to be used...

#include <string>

#include <iostream>

using namespace std;

int main (int p_argc, char* p_argv[])

{

string l_output;

l_output.resize(100);

char* l_temp = &l_output[0];

for (int i = 0; i < 1000; ++i)

{

memset(l_temp, 0, l_output.length());

sprintf(l_temp, "\r%i\0", i);

cout << l_temp;

}

return 0;

}

And this works perfectly as expected....

Please note, in the screen shot I had to increase the loop length on the second code example as it competed so quick I could not get a a print screen press before 1000 iterations ended!

I've not gotten to the bottom of this one yet, nor have I checked it out in G++ on linux or on older Visual C++ versions, I had VS2010 in use and spotted this going on... Annoying.

BTW - Yes, this is a good way to use sprintf and string formatting with a std::string.

---

Update - This does appear to be a bug, it seems the cout::operator<<(std::string) function assumes a return carriage, new line (\r\n) at the end, treating this as a "WriteLine", rather than an out string, and so you have to cout << stringValue.c_str(); to avoid this problem.

SkyPlayer Failing Whilst Debugging Code

I usually sit coding with some form of streaming media running on my secondary screens, notably I use BBC iPlayer, Channel 4 OD, ITV Player and Sky Player... The latter of which I pay a hefty subscription for the movie channels and everything.

Now, this evening is the first time I've tried to stream one of the films live, I usually download them to the SkyPlayer desktop application and let them run locally, but this often means a loooooong wait for the film to download (as my connection is usually very busy).

And I'm sorry to say, that playing films live, whilst debugging code is an utter shit.  The keep stopping and stuttering, distracting me from my code, and worse of all if they play they seem to get so far and then report an error that the media file is not available and give the obtuse "contact sky help pages"... which are no help whatsoever.

Now, the machine is a octo-core i7 (yes 8 cores) its got massive 16GB of RAM, its hefting two 24" screens off of a dual 470GTX SLi graphics solution... Yet it can't play a media file from sky, which its been playing no problem for the previous 15 minutes?  Just long enough for the film to actually begin proper...

They have no problem screaming the pre-film adverts to me, adverts despite this being a paid for stream already!

So, I started to get itchy and look what the service is doing, now I have a suspicion that the application running the stream (a silverlight application - urgh) is testing, looking at the system, for things it thinks might be trying to rip the film off...

And I think its decided that having Visual Studio open is one such situation.  I can't be sure, but everytime I start the film playing with VS open, after a while the player stops falls on its ass, arses around with a buffering page for longer than I care to think and then it says this stupid "can't find the media file mate" message.

Without Visual Studio running... It plays no issue... hmmmmm

I can't prove a link, but if I do, expect me to just cancel my sky subscription, I've already had a moan at them this month, see my earlier post, this is just another nail in the coffin.