Development : Simple Versus Complex

I have been reading some of Alan Turings works, specifically some of his work for the NPL in the late 40's and early 50's, before he went to Manchester in frustration with the NPL's lack of progress with his ideas.

One of the ideas Turing was pressing at the time was to do as much as he could in software, don't jump to hardware solutions; which at the time fixed the functionality of the machine to a single purpose; instead he wanted a general computing device he could put to any uses he desired.

Today the average programmer, hobbyist or indeed employed software worker, such as myself, don't have the opportunity to define new hardware for a bespoke task, indeed the machines we are driving are far more powerful than Turing ever imagined and we can do as Turing suggested, we can write software to get around an issue.

You need encryption, why pay to plug in a dongle doing this, when the processor you have is equipped to do the task for you, indeed some processors even have encryption instructions and powerful parallelism built into assist in such work!

So today's lesson is, take on board Turing's ideas, do in software as much as you can, even if you want to later move to real hardware emulate your idea now.  Here's a very good example:

nVidia's emulation lab is an extreme example, however, it is indicative of the kind of problems we are meeting today as older engineers are coming away from dedicated hardware devices (because their machines were too slow in the 1990's) and we have today's extremely powerful machines right there at our finger tips.

Why does this exist?

Last week I set about looking at a quite old system (circa 2008 it was last attended to) it contains a message passing system, which allows passing of messages (events if you will) between producers in any process to consumers in any other process subscribed to the service.  It did this with a windows service driver, which called down into a USB dongle to queue the message.

The consumers then polled the USB device (round robin style) to determine if the next message was for that consumer... You might imagine this was extremely slow.

It was also very complex, the code within the USB dongles was not mutable; being set in silicon; and the consumers could end up locked out of the service queue by a message being present for a consumer which was not able to consume the message.

This clearly needed replacing.... So, I had to look at it, a single afternoon, it was just extremely complex inside, so one had to take a step back, don't look at the contents of the code, look at the API, the functions being used....

The producer code relies on just three functions and the consumer relied on a thread safe queue which was woken up from a spin wait to call the registered consuming function... Not rocket science.

I set about writing new code with a very simple UDP server and clients, taking the message in, converting it to XML and posting it to the loopback IP Address of the master UDP listening port which then sent it out to the listening consumers on their loopback IP and port number.

Voila, in an afternoon, I had replaced five years (ish) of complex, hard to maintain code, with a working prototype of a solution.  And it dropped straight into place.  The dedicated, slow, old, hardware could be unplugged and the system just carried on as it always had.

The customer for this product is delighted to see it running faster, and the overhead of the hardware costs being removed.  All by simply doing in software something which was being offloaded to hardware in yester-year.

Administrator : Friend Still in Failure

Can you  hear a banging noise?... No... Are you sure?... I can.. Oh no wait, that's my head slamming into the desk.

A few weeks ago I posted about a friend for whom I uncovered a security loop hole for, the python script being run as root...

Well, it seems the same person, on their main "enterprise windows server" has been similarly hacked by the same chap, but the trick was this one.... http://megalomaniacbore.blogspot.co.uk/2016/04/windoze-security-loop-hole.html

Yes, a loophole I had mentioned and blogged about.

The loophole was exposed by a C# program the chap had written for them, I'm not sure what it did, but it was run as a service, and could spawn other programs under the same user, he'd written it so that it ran any program in a sub-folder then before he'd left he made it run a VLC remote desktop and another small program which opened a "Save As" dialog.

It opened this way way off screen, at something like location 20000 x 4500.  So the only indication it was there was a small twitch to the task bar, but he used the mouse to the browse the dialogue to the command executable and opened himself  a prompt as the administrator user.

I didn't spot this, I didn't even look at this machine for my mate, instead over the weekend I had a machine at home I was using as a network routing test, and I asked for some space on a remote server with a fat pipe to do a slow-loris type attack on my software.

I set up my end and set up their end, expecting to get something like 200mb/sec attacking me and swamping my little 64K buffer (I could then scale and tune my software to defeat this kind of attack, was the plan).

What I noticed was that I only got around 8mb/sec coming to me, so with my remote session I did a little digging and saw the disk activity was around 34%, with no-one in the office?!??!

The developer they'd let go was running a torrent service from the machine!  He was downloading and seeding several video streams of dubious merit and origin.

I didn't change anything, I just called my friend and explained... And I left it there, I'm yet to hear what was going on.

However, I think my friend will be having a word with the IT Admin he's had in place for nearing a month for not spotting that out of hours and weekends this persistent idiot is still infiltrating and using their systems as his own repository.

I've told them three times now to unplug everything and to put a secure router between their switch and the outside world, something brand new and only set up by them, pfSense perhaps, but they've not listened.

People : Email too Old?

Today I had the issue of someone calling me, repeatedly, and I asked them to simply drop the information in an e-mail (my well published e-mail - it's right there on the right of this page, and they had it right in front of them).

Their reply however stunned me.... "Don't you have twitter?"

My reply... "Yes, but not for this, email it me."

"Can't I send it you over whatsapp?" They persisted.

"No, just e-mail it me, it's a PDF or something yes?"....

They continued "So, how about through the cloud?"

"Just e-mail it me, as an attachment!" I was getting exacerbated with them at this point.

"But" they began "but e-mail is so old".

I felt like ripping my hair out, there were reasons for this request, firstly I could get their return information, second the attachment was over 35mb and finally I didn't want to have any complications about where to send the information.  They had my e-mail address right there, right in front of them.

The net result, so far, however is nothing has arrived, no sign of it... I'm tempted to phone them back and point this out, but I can't be bothered... Phones, like me, are so last century after all!

Health : Burnt Toast

Catching this one in the news this morning...

And I already know this... I already know Burnt Toast is a health risk, I remember being told by an Oncologist when I was about 15 that burnt toast was, in his professional opinion, responsible for the spontaneous occurrence of stomach cancer in some of his patients.

I remembered this, I remember when I worked for Boots mentioning this to some folks and their laughing.

I remember telling a house mate at uni and his sceptical look; but then I was rather odd, especially in the final year of uni... I used to cool my toast in the fridge for example (I like cold toast, okay.. nothing too strange about that!).

So, as far as I can remember this isn't news, it was news about twenty three years ago, but since then I've known this, about coffee too... I still drink it, as the oncologist I knew said, it either gets you or it doesn't, don't worry about a slice of toast; worry more about first and second hand smoke, diesel fumes and UV light.

People : Swearing Developers

I have to be honest, I swear... A lot... I sometimes do it knowingly, but there's also a little swearing leprechaun inside me.  My grand parents never really swore a lot at all, but my mother used to swear like a drunken sailor (i.e. a lot).  She thoroughly denies this now, but I assure you she swore a lot...

Consequently, I can swear a lot.

But this may not be a bad thing... For example, working with a development team as long as I have, you get to know people and you can express yourself, as I expect them to express their selves to me, if that needs a suitably short four letter inflection, so be it.

And it seems somewhere, someone, has decided this is not a bad thing...

This is lifted from BBC's Quiz of the week for 20th Januaryt 2017, and I have to agree, I am a straight honest person... But if you're being an arse, I'll tell you.

People : Donald Trump

This afternoon, in a few tens of minutes actually (by the time I schedule this post) there will be a new President of the United States of America.  A controversial figure, whom some might argue didn't win the election in a 100% clear cut fashion (his opponent - Hilary Clinton - won the popular vote, and then there's a little grey area with foreign involvement in the whole process).

But, I'm British, so I'm not too invested in who the man is, I am more interested in his politics, he's a half Scottish, through his Mother and has close ties to Britain.

We ourselves are not enamoured with the whole EU affair, and so trade avenues with the US may now open back up.  I certainly have a vested interest in the US economy as one of my pensions is tied up over there in a US fund, but for the tiny amount I put into that pension it's paying for itself, and depending on how things look in a years time, I may have to jolt things around.

However, we're left guessing about the next steps he will take, we can not tell whether he'll stick to all his pledges or promises.  He is however a business man, rather than a dyed in the wold politician, so he may have more rancur than previous presidents elect.

But politics aside the question today then is what will happen with the man?  Will, with protesters, detractors and so many unhappy voters in the US, both foreign and domestic, will the man survive?

This is a very serious question, one of the major points of out going president Obama's calls was for gun reform, however, everyone and their granny is seemingly still packing a piece in the US, even in liberal New York City & Washington DC.

So, will he survive the day?  The week?  The Year or his whole term?  I think that is an important question to ask.

As for the politics, we can but wait and see, though I believe the markets will heave this afternoon and though the seas may not boil and the sky may not fall, Donald Trump will be sworn in.

Development : Microsoft Tools Troll the Office

Today, I've been working on a brand new project, which literally I thought was parked just yesterday... My mistake it seems this project was expected to be being worked on.. A coworker is currently slogging through the same API but for a different (embedded platform) and it's taken him quite a while.

So the product owner for this new piece of software from me (which is to call XML Web Services through SOAP up stream and a hardware device over USB or serial down stream) was quite upset when they arrived at my desk and asked how their product was coming, and my reply was "Doesn't exists, waiting for the hardware".

After a short, and shirty, moment with them I assured them I would do all I could with the interface, implement ALL the XML Web Service calls through the published ASMX tomorrow... Which is today... They looked sceptical, gave me that long stare saying "no, no you won't"....

And of course I have!... Microsoft Visual Studio Troll tool award of the day, it is so simple to work with a published ASMX, just add a service reference to the URL and it generated everything for you, then tell C# you're using that name space and voila there it all is laid out for you on a platter.

I just finished this project, less than 5 hours after starting it, so have dished it back to the product owner to find the hardware, I know they're not going to believe how quickly I've completed this cycle on the development, not least because the other chap has been on it weeks and has only done about 60% of the calls so far... Yeah, I say... But he's not using troll Tools.

Sometimes, times like this in fact, I quite like C#... Don't let on, it's my dirty secret.

Administrator : Shredded Drive Bays

Yes we've all had to dispose of redundant data, be that drives or disks, I've even had to shred paper cards in my time... However, I recently came across a bit of a problem, an old Dell Equilogic was being decommissioned.... I had dibs on the unit for redeployment to a small development team.

Out of the Enterprise and into the shop floor, trickle down economics right?!  Right?!  RIGHT?!??!?

Nope!

The staff decommissioning said to one of their minions, remove the drives from this unit and put them through the shredder....

So he did... Literally, he pulled the drives in their bays and sent them off for disposal...

Yes, the CADDY BAYS!...

He was meant to remove the drives and put the caddies back into the unit, then shred the drives.

Now we have to find new caddies, on a zero budget, for about £18 a caddy... And we need twelve of them... So in thirty seconds this has cost about £216 hardware replacement costs, lest my time and the time of the poor guy who I sent scouring the place for these already trashed units.

The moral of this story, teach your staff what a drive is, what a caddy is, how a screw driver works...

Developer : How Big Is Your Floppy?

What are they teaching kids today?  I've just had the pleasure of talking to a gentleman whom has received a degree in computing from Stanford in the United States, I'm sure he paid a fortune for this education... He didn't know how many bytes were in 1.44 megabytes.

He literally said "about 1.44 million".  This was related to an old format of file being taken and converted for transmission over Wifi, the size of the data was relatively immaterial, but the source; historically speaking; was a 1.44 megabyte HD floppy disk, and though the floppy was long gone, the format of the file with it's limitation was not.

It got me thinking though, he was sat there, I'm sure earning more than myself, and he only had a jogging mental map of memory sizes, so I took a look at the Standard website, particularly the CS101 course... sure enough:

• "Byte" - unit of information storage
• A document, an image, a movie .. how many bytes?
• 1 byte is enough to hold 1 typed letter, e.g. 'b' or 'X'
• Later we'll look at storage in: RAM, hard drives, flash drives
• All measured in bytes, despite being very different hardware
• Kilobyte, KB, about 1 thousand bytes
• Megabyte, MB, about 1 million bytes
• Gigabyte, GB, about 1 billion bytes
• Terabyte, TB, about 1 trillion bytes (rare)

(Source: https://web.stanford.edu/class/cs101/bits-bytes.html)

Yeah, a terabyte is rare?... NOPE!... A kilobyte is not "about" anything it is exactly 1024 bytes, and a megabyte is not about 1 million bytes, it's exactly 1024 kilobytes, therefore exactly 1048576.  By using this "about" prefix they had made a megabyte sound 4.8576% smaller than it actually is, that's not insignificant!

And so forth.

This looks very much like an out of date page, but it is the first search result from google too!

Even wikipedia gets the right values before this page did!

Development : No Tests before a Prototype

I am NOT a fan of test driven development, does this mean there isn't a place for tests whilst you work?  No, it does not, I believe there is a place for tests whilst you are working, however that place is beyond the initial hard development and broad strokes of laying out the project.

When you first begin a new project there are so many things to consider, settings, scopes, layers, inherited domains just so much you need to physically take from the design pages through your teams brains and down to the keyboard.

I could even define this as a "pre-sprint" within Scrum to push out a product that the owner can then give you that first feedback on; especially if they are none technical and from this point on, from this first step you can cycle through Scrums into Sprints and with that change you can seed your development into more expansion and maintenance of code, when you do that, when you have something to test against then you need to add those tests and start to use them as a tool.

They should not drive your development, they should drive your keeping the developed work in order.  N-Crunch, N-Unit, G-Test... All these frameworks on certain languages (like C# and Java) work very well as you can reflect out the language, but before you can reflect out something with languages like C++ or C you are stuck over-engineering your tests before that first evolution of the project.

There is a name for this first evolution, it's called a Prototype.

Not many teams value a prototype, indeed Scrum itself never mentioned them, you are meant to jump from idea to stories to backlog to releasable code, and in my opinion this is not easy, it's not really communicative of what you are doing either, especially when the product owner is the only person within the development stack whom can redirect the team, but without that first tangible something many product owners can be literally lost.

Earlier this month I talked about the idea that a product owner needed to use the software life-cycle, an old idea, but a good one... Today I'm saying before you can really start to use inline tests within your IDE and before you can start to run scrums you need a prototype, arguably an even older development paradigm.

So what is my point?  Why do I keep bringing this kind of topic to the fore?  Well, simply put I believe there are far too many teams with far too many people in them not willing to push the envelop, whom are not willing to ask questions of the process they are following, not willing to bend or twist or shape the working environment to their way of working whilst simultaneously keeping up the bests practices those processes are there to enforce.

I suppose a Scrum Master should help with this process, but I find far few do, because the division of my problems with these Agile development mantra lie in the disjoint between idea and actually hitting the keys.  At times I see teams which are literally headless, they are a dozen monkeys typing a dozen keyboards, and they're being cheered on by the Scrum master, beyond those walls however they've no idea how it applies to the company, they have no interest or feedback on the return on investment that the Product Owner is all about.

A prototype, beyond a discussion, beyond the initial design, it can form the best kind of spring board and drive the best kind of Product Owner feedback down that chain, but it has to be created unfettered, unburdened by the micro-management of a scrum or sprint, it has to be created in a linear no-none-sense holistic head-space and used as a tool.

This is not being done very often in the teams I see around me.  Perhaps because it is thought of as old school, as a "has-been", however, if you look the hyper success of some products; say Minecraft; it maybe managed however Microsoft want now, it maybe teams running sprints, however Notch started it all with a Prototype.

People : Who is Robert Yates?

I'd love to know who Robert Yates is... I noticed a small bump in my viewing figures, someone had been viewing my site over and over with a source redirection of "protopage.com".

I've never heard of protopage, so took a look and found my humble blog here being a feed on the page of a one "Robert Yates".

Robert, thank you for adding me as a feed on your page, to reside alongside such illuminating voices as Dilbert is great to see, my only question is why?  What brought you to my humble abode?

Development : Scrum needs a Software Life Cycle

I'm trying to avoid sounding like a dinosaur today, but I have an issue with Scrum... I've been a team member and a scrum master, but I've never really been a product owner, until now... I'm currently in the middle of a large background turn around for my employer where a project which was largely parked and ignored for nearing three years has been awakened like some mammoth from the permafrost.

For the project in question though it had been frozen for a long time, the whole team who worked on it had been let go, I am the soul development survivor.  I therefore find myself the team member, scrum manager and product owner...

And so I'm being pragmatic as I swap roles and perform different tasks, I do have a tester with me, so I have to hold a scrum for him, and I've passed on several stories and he in turn has helped, from the requirements, produce stories for us to tackle head on.

The large part of the work however is from a document, handed to me, and I've been left to generate some movement.

The first thing which has struck me, and something which you never really see mentioned when you look at the Scrum training materials or concepts, is that the product owner needs themselves to have a Software Life Cycle.

They need to Analyse their requirements, to Design what the product should be then hand it over to the team to Implement, before seeing it gets Tested and meets their requirements.... Very much as I was taught the general software life-cycle in the 90's and used it thereafter... Long before Scrum was in vogue.

I asked a peer to review my thoughts on this, that we were driving the development for the requirements at hand with a software development lifecycle, which struck both he and I has "old hat" delving down the chain to the team (still myself, but that will change soon) as a Scrum & Sprint pattern.

Our conclusion is that, indeed a "Product Owner" when they come to think about "AS A <ROLE>, I WANT <FEATURE>, SO THAT I CAN <DO SOMETHING>" stories they are really unguided, that Scrum talks about the team understanding the requirements, but the methodology assumes those above have a good grasp on the requirements.

We went to far as to say that modern development of small light parts, or pages, within an app or web-page might never meet this dichotomy, that you need to guide the whole system and thought about the whole thing long before you come to write your stories, only a large system implementation; such as I am undertaking; might fall into this crevasse.

I can see how easily it might be for myself in this temporary triple role to lead myself off course, I can see how hard it might be for the Product Owner to express what they want, and I look at the Software Development Life Cycle and I think... "If only they'd mention that Product Owners should be along that road before they run into Scrum", some projects might run more smoothly, the tasks of Scrum Masters might be easier.

Developer : Being the Outsider

As someone controlling projects it's sometimes hard to get into the action with others around me, the best way is to talk to those people, to be involved to offer anything you can; perhaps not opinion, but fact to start off with.  Over time working with people this way I've always found you are invited into the more intimate meetings, the nuptials of the project, if you will.

This is a natural, human, process.  Great, or grand, projects such as Nasa's probes to the outer planets in the 1970's had much press attention, indeed some projects such as Nasa'a Magellen probe of the 1990's had press not only present, but directly interacting with the engineers and scientists (and then writing excellent books - "The Morning Star" by the late Henry S F Cooper anyone).

However, the mundane and private enterprise contains projects of a lot less public interest, indeed they are private interactions.  And if as a member of a company you've not yet had time to foster those friendly interrelations to gain constructive access to the meetings about a project you do find yourself as the Outsider.

You can hop and jump and try to peer through the murky world of hints and conjecture, but without getting up and digging into the project this can be hard work.  You may just want to help, but those inside the project might perceive you as stepping on toes, or going beyond your bounds of responsibility.

This is where the management have to see something, if you're already in a position to approach the project as a higher grade or manager role and offer yourself into the loop, that's easy and relates back to building bridges.  But if you're not, management have to recognise and apply you to the project.

Beyond team membership, stepped aside from the scrum master, and not overarching the product owner; to use the right buzz words, that is you have to literally be a fly on the wall and then slide yourself into position to help, rather than hinder.

How you do this is dependent on the people, you can offer code review, document review, hints or tips on hardware or architecture.  You can assist the team members or scrum master, even getting a round of coffee in can be that little bit which helps keep the folks on track.

But without yourself oiling the cogs of development with presence, approachable and an obvious willingness then you will likely remain the Outsider, you won't be on the ground floor for development meetings, you won't be in the loop with e-mail about project, you won't even know what is going on around you.  If you end up in this situation, you need to address is up the chain, to push for somewhere to help yourself fit.

You always need a good manager above you to help with this, and in return you need to help them in order to help yourself.

Administrator : Fired Employee Stole Software Serials!

Today I've had to deal with a security issue for a friend, a certain department manager had a machine with a set of serial numbers for each of his developers tool stack, quite expensive stuff, these were stored in a text file on his desktop in a folder called "serials".

The problem?  Before Christmas an individual was asked to leave the company, his parting gift?... He stole all the software.

They knew this because he got a little drunk with a common friend of the manager over New Years and boasted he'd be starting his own company with the same software stack... We're talking about £4000 worth of software.

The machine the chap was working on (windows 7) was pretty locked down, he could not  install, nor run a cloner, nor could he add a drive or boot from CD/USB or even get into the BIOS settings to take the software directly off as it was installed.

My friend, investigating, suspected that the chap had simply taken the serial numbers, the manager frankly denied this.  His machine was checked, no-one had been accessing it other than him, there was no file sharing set up and it too could not be booted into anything other than the Windows 7 installation on the machine....

There was no-way, in the managements eyes, that the software could go missing... My friend suspected different... And so co-opted me to help, getting a copy of the disk I was not able to log-on, but I could see a whole bunch of scripts run at start up, whenever ANY user logged onto the system, they ran a series of scripts, one of which was a batch file (slaps forehead).

This batch file was stored as a regular file in a folder "C:\devscripts" and could be written too... I had a hunch this was where something could go wrong, but couldn't initially see how, the file looked unchanged... But then I noticed a hidden folder... a git repo metadata folder... in the same directory.... It seems the source of both this folder and the file was a git repo, so updates to the devscripts would be stored and commited to the git repo... and pulled to each machine every-reboot.

But the git-repo was open for ANYONE to commit to!

The developer had simply, within the massive cloud of commits, committed a new batch file to the repo.  This new script started the python interpreter with the command line "-m http.server 8080 C:\Users\<manager>".

The development machines used port 8080 for the remote debugger, and let this connection start without warning the user that a new firewall rule was required.

The developer then simply used "wget --spider" to pull all the files from the managers desktop, this included a bunch of documents about staff performance and of course the serials folder in it's entirety.

Once he had all these files, just before the festive break and his departure he committed the original script again which removed the starting of python.  He did this in the midst of the sprints to the break no-one noticed that their commit numbering had slipped by a place!

If he had reverted the repo he'd have needed the managers approval, but as it was the script just went from green to a yellow to a green against their sign off system, so no-one paid it any attention that the revisions had slipped in date & time.  And that the sign-off system didn't treat the signature of the commit as being different was a bug in the system.

So the lessons to learn...

• Don't let batch files start as the root user on a machine, EVER!
• Don't update said files from a public or even open internal source
• Don't just ignore any subtle changes (like the commit number order changing) when you observe them!

And when you fire a developer, fire them!  Don't let them sit there with anything connected to your network, you just fired them!  They'll want revenge!

Administrator : Show Network Abuse...

As a network boffin it's always very difficult to express to every possible audience quite how busy & complex a job you have, not least in today's environment where you have wired and wireless connectivity from every imaginable device, uPnP, SSDP, ARP, netbios, Active Directory, DHCP, DNS, UDP, TCP... It's a plethora of fields you have to take collectively as knocking on the proverbial door to your network adapter.

Recently we've employed the excellent Solarwinds tools as a method of network node and inter-connectivity debugging, and as much as I enjoy presenting the truly informative flow-charts and information they don't show quite how huge the amount of data and number of packets flying around is.... I therefore set about visualising and displaying just such a situation to an audience...

A Ubuntu VM, with the I3 desktop and a little time later.....

cd~

git clone https://github.com/the-tcpdump-group/libpcap.git

cd libpcap

./configure

make -j4

sudo make install

Then...

cd ~

git clone https://github.com/the-tcpdump-group/tcpdump.git

cd tcpdump

./configure

make -j4

sudo make install

And now, I could... "tcpdump"... to just spew everything into a terminal window...

Open another terminal window and I could "tcpdump | grep 'facebook'" and see how much time people were flying off to facebook, the beauty and simplicity of the i3 desktop along with the raw output of tcpdump soon conveyed far more stark a message from a monitor on my desk than any e-mail could or diagram could communicate.

Just the TCP packet headers were enough, without prying into what was being sent over a company network, in company time to facebook... and ebay... and whatever else... 

Using egrep I could dig even further into the headers and get very specific, laying out the desktop with i3 just further emphasising the flow and pattern of usage... Anonymize it and I'd have a decent techy screen-saver too!