Sunday, 21 May 2017

1000....

Welcome to my 1000th post... I'd like to thank everyone who appears on these pages.

You are one of nearing one thousand unique visitors each day to my various posts on this blog...

Now, if only you all left a tip... Hmmm...

Friday, 19 May 2017

Football : Too Arrogant to use the Technology!

I enjoyed this from the BBC, however, I have to say Mr Nevin uses sage words almost throughout.

Save for when he says "Too Arrogant to use the technology"... Quite who he is referring to is not clear, UEFA, FIFA, the FA?... Perhaps all of them, whomever it is (or was) we're yet to see this technology, until 2018's World Cup...

Take a look yourself.

Monday, 15 May 2017

Development : The Cathedral & The Bazaar

Recently I had to explain to a none-developer how we had been working on a project, that is to say he and I.  I explained that our common boss worked very much in terms of the Cathedral, everything is big, grand, old, with set sources for supplies and almost dogmatic adherence to their ways of working.  Whilst he and I had been working, with extraordinary success, in a much more agile manner, in the manner of the Bazaar.

Anyone could bring items to us, he could filter some queries, I could filter some others, between us we then worked in an Agile manner to achieve a result.

I myself ran the development, and I pushed builds, lots of builds, releasing at least twice daily.  Any problem was picked up, worked on, solved and the next build had it, leaving a train of progressively better builds step by step, until we sit today with a fully functional system and we await input from multiple other sources.

The main source for things we are awaiting is the Cathedral, we have waited a long time, seven weeks of hard work by ourselves far out stripped the expectations of the Cathedral model of working, and once we superceded that model it was too heavy, too slow, too stiff to catch up with us.

The net result is somewhat of a pocket of confusion, work was demanded, it was delivered, yet now it sits unused.

The bazaar delivered, whilst the Cathedral struggles.



("The Cathedral & the Bazaar" is a famous musing upon software development from the perspective of old slow methodologies to the exploding Agile methodologies which suffuse the Linux development and open source development movements).



Saturday, 13 May 2017

Story Time : "Younger Man" with Sleep Apnoea

Not dissimilar to the description J K Rowling gives to the character of Uncle Vernon in the Harry Potter series, I am a "Big Beefy Man" with "not much neck".

As a result last year it became blindingly obvious that my waist was expanding, I also personally noticed a drop in my day-time alertness.  Initially I would like to point out that I have always been a night owl, and it was not uncommon for me to stay up late and get stuff done (especially programming things) and then sleep later.

This fit well with my employment offering flexible start hours too, and so happily for very many years, I got slowly fatter and fatter, sitting down and eating too much, and I slept later and all was merry.

Except, it wasn't... I suddenly noticed I had become extremely moody, irritable, bad tempered, my concentration was dropping and I found it nearly impossible to remember new things.  Initially I bit back my bad temper, and to compensate I started to use a series of physical notebooks and then scheduling apps to help abate my failing memory.

This crooked self-dillusion carried on for about four weeks, the wife later told me I was a nightmare to live with, however at the time I could not explain the reasons for the change, I could think I didn't want to act that way, I didn't want to forget my lunch, or eat junk, or be a bastard.  But I was being, and there was nothing I could seemingly do about it.

So, I took a few days to think about what I was doing, what had changed, I was very tired, I had to admit.  I kept going though... Kept going, caffeine, pushing black coffee down me, and sugar, lots of sugar.  I never felt I was eating well, but I needed the energy, without it I faltered, stumbled and one day I said "no caffeine today, lets see how I get along".

Driving home one moment I was at red lights, a blaze of other tail lights a head of me, the next blink, I was alone... Sat at the lights still, no-one around me... I had fallen asleep at the wheel.

I literally pulled over and slept there and then.

I contacted my GP, and said "I think I have sleep apnoea", this notified the DVLA, I suspended driving and got one with treatment... Unfortunately this took sometime, so my coping mechanism in the meantime was to sit up in bed more, and to fit a black-out blind to the window.

This alleviated symptoms, but I knew I wasn't sleeping deeply, a short time later and a night of sleep monitoring it was confirmed, I have obstructive sleep apnoea.

I thought this was because I was fat, because I was getting older.  But it seems thats not just it, I am large, and all my family are.  However, the tiredness makes you want to eat more, the BBC has recently covered this phenomenon more than once, and I have to admit it was nice to hear.

I was issued with a CPAP machine, was immediately sleeping so much better.  It was like night and day.  And for the last year I have enjoyed this machine immensely.

However, the straps for the masks go slack over time, as they go slack your sleep falters again, as a man if you have too much facial hair air escapes the mask (have a beard but very short trimmed it's not a problem), far more of a problem is how the mask  seal skirt wear and then the straps go slowly slack.

I would have to argue that the company making these (Phillips) could be held somewhat to account for the latter, as it seems to happen at a predictable, almost designed rate, however I have to concede that the material used has certain requirements and the felt like vencro fastening straps with their slight elasticity are pretty good, but the continual re-stressing of the product does result in quite quick deterioration in performance, even before you take into account the biological factor (sweat & skin cells, and having to then wash the straps).

However a year on, I'm not wondering whether the machine is enough for my condition, I've noticed more than just slack straps, more than wearing seal skirts, more than just a bad nights sleep that more days than not I am quite tired of a day.

This is not everyday, but perhaps two days every fortnight or so, there's no obvious reason for this biologically, environmentally there's no changes either.  I have to admit to slightly more stress in my life at present, however pressure or stress have never stopped me sleeping before (a side effect of studying the martial arts - if a bloke bigger than me can hit me, a bit of code not working in the office is nothing to sweat over).

I will have to discuss this with my consultant in early June.

Being an every inquisitive soul I have started to look at other factors into why I might have apnoea whilst my brother (a close body match to me) does not, and as far as I know no-one else in my family - despite our having a string of heavy snorers - doesn't have apnoea.

One thing I mulled over, before doing some reading, is that during my mid to late teens I had orthodontic treatment, I had four healty pre-molar teeth removed and the two following molars behind these pulled forward.

I've just read, in two sources, that these molars play a role in strengthening the stiffening the sides of the soft pallet, and they also provide a framework onto which the throat muscles structure, then of course the net result is that your jaw is shrunk, leaving less room for the tongue.

I have to be completely honest here, if I let my tongue rest in as natural a position as it can, as easy feeling as it can be, it sticks out over the end of my lower front teeth.  But sits slightly behind my upper bite.

I have the distinct feeling that if I had not had this procedure, yes I would have slightly wonky front teeth, and yes there may not have been space for my wisdom teeth to fill in but my tongue would fit inside my teeth.

As a consequence of this poor fit I have to draw my tongue slightly backwards, with a kink mid-way, to let it sit behind my lower teeth, add the very nature of the action of the flesh being slack in my apnoea and this is a recipe for real trouble.

Having made mention of my weight problem, I did grow whilst without my CPAP machine, I went up to 21 stones around July 2016, by December I was 19 stones, and now today at posting, I am 17 stones and 9 pounds, you can go convert that yourself.

I'm still big, but I feel so much better, even with a few crappy days sleep I feel so much better.

Coming up in June, lets see if my consultant agrees with my orthodontic question, and if I need anything corrected or improved with my CPAP machine.

Wednesday, 10 May 2017

Development : Python, MySQL and Protocol Buffers Build

Today I've come to a totally virgin installation upon a server, this was for a work group I've got to head up whom are looking at pushing MySQL with Python.  And things initially went wrong...

I stipulated they had to use Python3 and thought everything else would be fine for them to install with Pip3, so...

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3
sudo apt-get install python3-pip
sudo apt-get install mysql-server

Everything looked fine, their user could install packages through Pip3 and they had Flask and a few other things and were flying.  I used the mysql client on the command like to test I could add a few items and also ran...

sudo mysql_secure_installation

To do the basic hardening of the database, so everything was fine... Right?....RIGHT?!?!

No, not exactly.... "I can't get mysql.connector".... Came the cry.

And they were right, it reported a myriad of build issues and could not install.  I took a look... NIGHTMARE!

It appears the installation of mysql.connector via Pip3 depends upon Protocol Buffers from google for the latest version of mysql.connector... Which the Pip install didn't sort out, at least not easily... Luckily I run a whole gaggle of virtualized machines, so I could quickly spool up a new instance and try a few things out...

This is the script I came up with....

cd ~
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -y python3-pip git autoconf automake libtool curl make g++ unzip
sudo pip3 install flask
hostname -f > hostname.txt
sudo apt-get install -y mysql-server
sudo mysql_secure_installation
sudo ldconfig
cd ~
git clone http://github.com/google/protobuf.git
cd protobuf/
./autogen.sh
./configure
make
make check
sudo make install
sudo ldconfig
cd python
sudo python ./setup.py install
cd ~
sudo ldconfig
sudo pip3 install mysql-connector --install-option='--with-protobuf-include-dir=/usr/local/include/google/protobuf' --install-option='--with-protobuf-lib-dir=/usr/local/lib' --install-option='--with-protoc=protoc'

Lets go through this step by step...

cd ~
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -y python3-pip git autoconf automake libtool curl make g++ unzip

This is just the basics, so our installation will now depend upon git, autoconf, automake, libtool, curl, make, g++ and unzip, most all your standard distributions will contain packages for these, we use -y just to skip any permission requests.

sudo pip3 install flask
hostname -f > hostname.txt

The next step is simply a couple of items for our project, we're going to use Flask to provide a restful interface, and the "hostname.txt" is simply to remove our need to call "hostname" again later.

sudo apt-get install -y mysql-server
sudo mysql_secure_installation

Our next step is to install and secure the MySQL service.

sudo ldconfig
cd ~

Generic code now, to simply reload the library list and change to the home directory.

git clone http://github.com/google/protobuf.git
cd protobuf/
./autogen.sh
./configure
make
make check
sudo make install

This is the build of protocol buffers from google, so we pull it from their github repo, we move into that folder, prepare and configure the build, then make the whole build.  By far this is the LOOOONGEST instruction, on a single core 1GB equipped virtual instance this took around 45 minutes.

Once complete we simply need to reload the libraries again...

sudo ldconfig

However, protocol buffers are still not installed within Python, so we are still in the "~/protobuf" folder, we now need to go deeper, into the python folder and perform the setup installation under python...

cd python
sudo python ./setup.py install

When complete we again need to reload the libraries...

sudo ldconfig

And the final, secret sauce, is to actually install mysql connector through pip3 with protocol buffers...

sudo pip3 install mysql-connector --install-option='--with-protobuf-include-dir=/usr/local/include/google/protobuf' --install-option='--with-protobuf-lib-dir=/usr/local/lib' --install-option='--with-protoc=protoc'

This is a single command, spanning one single line.

And voila, once complete you get to use mysql.connector in your python code...

import mysql.connector
import gc

con = mysql.connector.connect (user='whatever', password='something', host='localhost', database='yeahyeah')

con.close()
con = None

gc.collect()

You can find out more about why I nullify and garbage collect a connection in my previous post.

Monday, 8 May 2017

Software Development : Get Constant in C++

What does "const" mean to you?  Does it just mean a value can not be changed?  If so, you may want to read on!

Const is one of those things, which though not unique to C++ is given more meaning when you leverage C++ fully, const allows you to not only define a value as invariant, but also to instruct other programmers coming to your code how a value should be used, how when it is passed as a parameter it should be treated and ultimately how to protect data from needless alteration, de-synchronisation or simple corruption.

Const is therefore your friend, and if you've come to C++ from C, Java, C#, Python or one of the other myriad of languages which don't treat const with as much relevance as C++ you may want to read more than I can say on the topic. 

Bjarne Stroustrop (the inventor of C++) and other authors on the topic (notably Scott Meyers & Herb Sutter) explain in much more detail than I ever could, but for brevity here are two examples of const from my own coding standard which I implore you to digest.

1. Initialise Const Values in the constructor ONLY....

Any instance of the "Ex" class can now access it's "Pi" value, and we communicate to all viewers that the value is a constant and only edited in one place.

2. Where-ever possible pass values as constant references....

We know for certain that the "p_Radius" parameter being passed is NOT to be changed by the function, this is important when you are thinking about letting code document itself, and easily accomplished with "const", and especially annoying when using a language lacking const!

Whatever you do, use const consistently, this needs no specific explanation here, however if you begin following my second rule, you must stick to it!  Don't change half way through a program, if you import a third party API which is not using const in the same way, abstract that third party interface away with a series of your own shims, to make it correct for your usage (the compiler will optimise all these layers away) to leave you with consistent and more easily maintainable code.

You can find must much more about const-correctness elsewhere, I'd start here.

Thursday, 4 May 2017

Software Engineering : My very bad way to set up makefiles

I may have wrote this post, but I'm just being lazy... Read on.

I've made other Boost videos, and other codeblocks videos, and I've always assumed that the user at the other end was well aware of how to compile & build code.

However, it seems there are lots of users whom are not aware of how you actual create a program from source code, at least not with C++.

So I'm going to show you the absolutely most wrong basic way of creating and using a makefile to perform a build...

The first step of C++ program creation, is that trivial matter of creating the source code, easy right?... Yep, so lets skip that step, and go to when its once  and we want to pass the code through a compiler, the compiler turns the source not into a program, but into an intermediary format called an object file or object code.

The compiler then leaves the process and another program takes over, this second program is known as the Linker, and it takes the object code and links it with all the system calls and other libraries you are wishing to use.

There are other steps involved in the process of building a program, however, compiling & linking are the two main steps; when I began programming you were made very much aware of this by the compiler and linker generally being such large complex programs and the libraries being so big you had to physically change floppy disk between the different steps of the process.

This process was also taught to me in school & later college, it was common practice to understand a compiled program was actually compiled then linked.

Unfortunately, this knowledge seems to have fallen in to the way side, it is of course more important to learn how to program than be bogged down in minutia, however, once you have learned I strongly believe you should expand your knowledge and really have a holistic knowledge of the craft of programming.

So, back to our program, the make file, what might they look like? Well they are simply text files,  which contain commands very similar to bash script... We call this make file script by passing it to the "make" program, you can get make for Linux, mac or alongside MinGW for windows.

Lets start writing a make file for any basic C++ program, which links the Boost libraries we've built somewhere on disk, and which looks like this:


#include <iostream>
#include <string>
#include <boost/filesystem.hpp>


int main (int p_argc, char** p_argv)

{
     if ( p_argc > 1 )
     {
          std::cout << "File or Folder ";
          std::string l_pathString(p_argv[1]);
          boost::filesystem::path l_path(l_pathString);

          if ( boost::filesystem::exists(l_path) )
          {
               std::cout << "Exists!";
          }
          else
          {
               std::cout << "Not Found";
          }
          std::cout << std::endl;
     }
}

Save this as "main.cpp" and then define a variable called "FILES" into which we place this filename.

FILES=main.cpp

We want to start our make file with a variable which contains the compiler we wish to use, for Linux on my test machine I'll use "g++".

CC=g++

So now we have a value called "CC" which is a string of characters with the value "g++".  The dollar sign indicates that "CC" is a variable within the script, which we might want to use later, I use these kind of variables throughout makefiles so I can place at the top any paths or references I like, or I can alter them when I move the makefile from Linux to say Windows, without needing to change hundreds of places throughout the script, I can change just one location.

I find this working with strings is 95% of what working with makefiles is about, next in the file we then define the strings for where to find both the boost headers and the binary libraries are...

BOOST_INC=/home/xelous/boost
BOOST_LIB=$(BOOST_INC)/stage/lib

Notice, I create the root folder for boost as one variable, and then in the next line I use that variable to extend into the "stage/lib" folder, so I never duplicate the root folder for the boost libraries.

We can now define at least our build command, slightly different the command is a verb, in this case "build", which is the command we will use here, but which is also the default.  If you save this makefile as "makefile" in a folder, then navigate your console to that same folder and type "make", the make program will look for "makefile" by default, find it and build the "build" verb by default!

build: 
   $(CC) -I$(BOOST_INC) -L$(BOOST_LIB)

So, what is this command doing?  Well it is saying to use the compiler, with the Include folder for boost and look for libraries in the boost library folder we've set.


The "-I" and "-L" parameters are the same parameters one would use on the command line for "g++".  This command therefore equates to "g++ -I/home/xelous/boost -L/home/xelous/boost/stage/lib", but I think you will agree it is somewhat easier to manage.

We also need to tell the program to link against some libraries, and that we want them linked statically in this case.   So our next variable will be called "CFLAGS1" to pass the Compiler flags to the command.  Back at the top of our make file I therefore add....

CFLAGS1=-fpermissive -Wl,-Bstatic

This file tells the compiler to be permissive of some common errors and treat them as warnings.  You can use whatever parameters you wish for your compile, the flags available and their meaning are quite varied, however you will need to use a second set to within the build command, so lets define that now....

CFLAGS2=-static-libstdc++ -std=c++14

The second tells the compiler and linker to link against the static versions of the libraries we select later, you will need to look at the boost invocation pages in order to go back over our previous posts and build boost static and take advantage of this option.

I'm not going to explain static linking here, nor any of the other switches, available.

Now we're linking the C++ standard library, and boost, statically we need another variable with the list of libraries to link, on the command line to g++ this is with the "-l" (lowercase L), except with we are going to make the program threaded, so we can set up a new makefile script variable called "LIBS", into which we set the libraries, like so.

LIBS=-pthread -lboost_signals -lboost_system -lboost_filesystem

We now have the compiler, the folders, the libraries, the flags... What about the target file for the build?... Well I want it to be an executable called "
fredrick".

TARGET=-o fredrick

Now we need to put this all together into a command, just running the script with the variables does nothing, we need to define a verb... Lets call this "build" a verb in the makefile has to have a colon following it...

build:

And we can tell make what to perform for a build:

build: 
   $(CC) -I$(BOOST_INC) -L$(BOOST_LIB) $(CFLAGS1) $(FILES) $(CFLAGS2) $(LIBS) $(TARGET)

We might also want to clean the project, so we'd want to delete the target file maybe, by adding another verb:

clean:
   rm $(TARGET)

We can then perform the call "make build" or "make clean" in the folder with this makefile and perform our build.

A bit rough and ready, but it works to get you up and running.