Wednesday 30 January 2013

Tester Communication


Understanding the difference between a fault, and a feature, is important in software development.  For most serious programmers we can joke that a bug is an unintended feature or even more honestly a cock up.

But sometimes you have to really listen to the person reporting a fault, to empathise with them and to get into their shoes.

This relating to your customer is a real big theme at the moment, as we're on deadline for today (in fact Friday, but we're saying today to put that extra spring in our steps) to deliver a really big system for a show, so now only has it got to work flawlessly, but it has to be in a saleable distributable situation.

For this we have a league of testers working live on the system, so as fixes feed out to the lead tester he does a generic test and all the automated stuff, and then he hands the builds over to different people to try to break them.

And today, I had a major call that one of my systems was failing soak, it was "leaking memory" was the report.

I have since last night spent perhaps 8 hours trawling the code, doing builds and tests and checking, there's nothing being wasted, nothing being unallocated, nothing being allocated and then abandoned, it is in the technical sense not leaking memory.

So, I've puzzled over this, and in the end gone down to talk to the testers, I've asked them to show me a machine, they happen to have one which had been on all night and it had a huge amount of memory being taken by "your application".

It was the logger, the log of activity, the debug log which should be turned off was still on, accruing string after string after string of text, nearly 500 megabytes of it over night in fact.  Double click to clear the log, or better yet turn it off, and no "memory leak".

What had happened was this user had heard of "memory leaks", and knew that the phrase was associated with memory being allocated that they could not account for.  But they'd not thought about what the system as actually doing, they had reported a "memory leak" and caused other semi-conscious technical users to panic, I myself had panicked.

Let this be a lesson learned.

Monday 28 January 2013

Bad Programmer to Customer Communications


There are five words decent application developers don't use when talking to their customers, these five words are...

"I'm not having that problem"

Because, not only are these self defeating words, but the customer has taken time to relay feedback to you, and clearly they are having a problem.

Now, if the user is a technical user saying these five words is essentially the kiss of death, you'll be instantly and irrevocably branded a dundard, if they're not a technical user you'll be branded an unhelpful fuck-tard.  But this impression you're giving your end user or customer is only one part of the defeat you will suffer.

The other part of it being your personal acceptance of fallibility.  I myself had a major issue with this until perhaps two years ago, I thought code I wrote was infalliable, I thought it was great, despite projects going over span and debugging taking time I always thought thee was a way to point the fickle finger of blame elsewhere.  And you know what, sometimes there's not, sometimes you've just done something wrong, and even if you've not and the starting point causing the whole problem is nothing to do with your code, it was your code which didn't anticipate and deal with the situation.

So folks, lets man (and woman) up here, lets be honest with ourselves and our customers, if something has gone wrong, take an interest in your customers plight.  And even if you're too busy working on actual problems to give this current issue time, make a note of it, e-mail yourself, set a reminder to come back and take a look and feed that back to your customers.  It might not improve your throughput, it might not actually solve a problem, but if a customer knows and feels you will get around to something and you will tell them, then they're impressions rise...

Wednesday 23 January 2013

YouTube Irritations

I've come to the conclusion that managing and working out what your comments and commentary gets replied as on YouTube is a nightmare.

Google have recently changed the whole YouTube experience to be more Google+, rather than its own stand alone service, and I for one, don't like it... There are a fair few obvious things missing from the mix...

1. When you comment, and someone replies, it comes into your "inbox" that you have a reply, and it shows their comment regarding your input, or trolling... But you can't see your original comment, sometimes the replies you get are months after you made the comment, and when you comment a lot its hard, if not impossible to track back what you actually posted originally to garner the attention of this reply.

2. You can't get to your inbox easily, you used to be able to just click on account and go into Inbox, but right now you can't.  I refuse to sign into Google+ and it bitches at me for not doing as they want me to.. But you have a right faffing job to get into your Inbox.

3. You can then track a comment as "Linked Comment" if you follow it back from your channel listing, so you hit your account, top right, go into "My Channel" and there you can click on the comments you've made and hence get back to the comments you wanted to read as the API actually puts them up as "Linked Comments" specially bringing them into view...


So here, from "my channel" you can see the posts and go back to see what the comment refers to, but not from your Inbox.. Gah.

All in All, for such a popular service I think YouTube needs a little loving its awkward with this attempt at them linking it into Google+.

I also have an irritation with blogger, when you paste in from NotePad a wall of text, it adds a <BR/> on the first line, leaving a gap at the top of posts, a gap which was not there in the compose window, you have to go into the HTML window to remove that... annoying... Everything is annoying today.

Tuesday 22 January 2013

Programming - Divide and Conquer

Today has been one of those days where I have gotten my head down and done lots of code, unfortunately I can bring none of it here to share with you lovely peeps, perhaps I could look at some of the reflection (of images) work I was doing and post an update to my image processing labs from last year....

But, I have literally just this second had another interesting conversation, another developer in the same department as me is working on a project all of his own, he's implementing it all himself, selecting all the libraries and all the facettes he wants to bring to the party, it is truely and in all senses of the words "his baby".

And he's moaning, he's moaning that things don't work, that he's spend ALL day looking for the source of one exception and spend hours the night before looking at one piece of formatting, and complaining how shit it all is... And I want to shout...

"BUT YOU FUCKING MADE THIS YOURSELF!"

He's fallen into the true trap of the over confident, he's taken a managable task and thinking he knows best he's implemented it all in one big go, rather than pace it or do it over the months he's had, he's basically sat idle and been thinking "Oh that job I got on is a doddle, it can wait..." and clicked on youtube and lol cats et al.

Well, delivery time is nearing, and he's rush rush rush up the product rate... Check in, check in.... No, one mass check in, so he has no structure showing in the revisions, he has no progress, there's just this big black hole of three odd months and then code poking out all over and its falling over left right and centre.

Folks, don't do this, let me tell you from long hard experience, break a task down, do a function at a time, prototype things, and when you've got them working DO go back and review them, break them down into functional nuggets, put a comment on them (I am very much in favour of comments, unlike some other programmers out there) and remember to keep on track don't get distracted.

And if, like this fella, you don't know 100% quite what is required, and no matter how you've asked you're getting no apparent input, then write it all down, or at worse mock it up in code, and run is past people.  Do not just wait until deadline and then try to hand over a god awful mess of code.

Monday 21 January 2013

Snowed in Britain

So its snowed here in Britain, it had threatened last week, hence I did 5am starts to avoid getting stuck.  But last night I could simply not get any quality sleep, so I opted to stay in bed.

Big mistake, I ended up having to dig out two foot drifts, and three foot troughs, up the entire back lane.  You might remember I mentioned before I live down an dirty track off a country lane by farm land.  Anyway, it was thick with snow.

Worse than that though the neighbours, despite my asking them not to, had been going up and down with their Volvo making the track way into ice, and I mean going out for trivial things, like a single loaf of bread, or six bottles of lager... and a magazine... And no, not all in one go, that was three different trips.

So, a good six inches of snow turned into ice, and then two more feet of snow have fallen on that... They're now stuck.

I however, am at work... The reason being, I am not so fucking foolish, but also carpet... Old carpet is my secret, put it under my front (drive) wheels and away I went, into the troughs I dug... If I got stuck, out I got, carpet in front of the tyres, roll on, and take off again...

I got to the road, after digging and swearing after about an hour and 45 minutes.  I packed my shovel and magic carpet and set off to work on the worse back roads I've been on in years, they were white out in so many places, with black ice beneath.

As it ease and I got onto the A roads and travelled routes into the city, but I had to boggle at the city dwellers making life hard for themselves, morons driving around still with the snow all over their car... You could not see their lights, you could not see their faces, just sweaty little mole noses pressed against glass trying to drive...

Its not snowing right now, the air and weather has lifted, and if they cleared their cars driving would be safe.  But they've not, there are so many wondering around with rooves covered in snow and the heater on... the result.... the moment they brake an avalanche takes over their windscreens and they have to slam their brakes on, its so dangerous.  And all because they don't have the patience or brain cells to clear their car.

I cleared mine, I cleared the lane, I got out... Good luck to the other people trying to clear a way, but these morons still covered in snow... Gah.... and one of them "Baby on Board" in the back... these morons are breeding!

Thursday 17 January 2013

PC Overhaul


I've decided that with my main PC being down and out, I'm going to refurbish it.  Its two years of age, and though mighty powerful has come physical problems.

You may recall I fitted silent fans to the system, well one fan system which was never silent was the power supply unit, so I am looking at replacing that, I bought a quite powerful high spec unit (an OCZ 1000w unit), but I bought it for my prior machine my old Core2 Quad, apart from the case the power supply was the only unit I carried over into my Core i7 build.  So its time for an update.

I'm looking for a modular, constant, power supply giving a constant 850w or more, I'd like 1000w for surety and the possibility of adding a third graphics card into my SLi configuration later.

The other problem is, though I added the fans, and sorted the air flow optimisation it seems I also optimised how much dust would get pulled through the machine, so I'd like to look for some micro-mesh screens or filters to go between the case and the fans to filter out the junk on the inbound side of my air loop.  The trouble is, this might increase fan noise once again.


It'll also be a good chance to look at my storage options and update some of my drives, not least as I just had a Hitachi 320GB hard drive (stamped on top with the date of Oct 2007) die on me, I don't know whether it was taken out as part of the motherboard going down, or it itself contributed to the motherboard failure, all I can tell you is plugging it now results in hearing it clunk and it get very very hot very very fast.  Its on my desk wrapped in a big red plastic bag labelled DO NOT TOUCH ON PAIN OF SMOKE ALARM as we speak.

I may look at two small (32gb) SSD drives, as they're sub £30 now.  And just put the OS on that.  Then add a larger SSD drive to host games on my Windows boot, and add other hard drives to host the Linux portion of my data and virtual machines.

I do have dual USB 3.0 external facing ports on the rear of the machine (when connected to the motherboard supporting them) so I may also look at a pair of external USB 3.0 caddies, or purpose built drives, to hold my virtual hard disk images.  This is also the new part of my way of working, I'm going to start separating my data from my programs more thoroughly, I'm often adding new virtual drive images to my Virtual Machines.  You can even mount the same Virtual Disk Image (though not at the same time) on two different Virtual Machines, so I can create a new NTFS drive, create code on it on Linux, then check the same code cross compiles correctly on Windows.  A very useful trick.

Also backing up a single (or multi-part) VMDK file is easier to manage than having to back up the individual files being controlled for code, obfuscating the work I am doing by an added layer on my subversion server.  Using an encrypted virtual disk also means I can opt to carry the whole of my work around on a virtual encrypted compressed drive image, and keep working from any device mounting it, rather than worry about multiple files or backups or importantly fragmenting my work across multiple devices; how many times have you changed a file on one machine (say at the office) think you've submitted it to your server, then gotten say home and found its still on the machine at work, and you either re-do the edits at home on your working copy or you leave the work until the morning?... I do that on occasion and it annoys me as dead time.  Now, carrying my whole virtual drive, and leaving a back up at work, and a back up at home, I can sync the two easily and bring hundreds of thousands of code changes/work alterations with me wherever I am without the risk of missing X file of Y changes... There can be only one changed file, only one latest... the oldest dated disk image, simple.

So, those images on external USB 3.0 drives... I sense a plan coming together.

Wednesday 16 January 2013

Sustainable Office Buildings


I find the prospect of carbon neutral buildings interesting, but not as interesting as office buildings heating and powering themselves, not least for small start up companies; which we have far too few of in Britain.

I read last month a book regarding Sustainable Refurbishment, and was impressed with some of the ideas and case studies it presented, but none of its content came close to this idea, which though likely not to happen, is a pretty good idea.

http://www.bbc.co.uk/news/business-20965207

Annoying Google Banner

I've just opened google for the first time this morning, and noted that their title has changed.  Its a small game you can click on and get a chance to clean the ice on the rink as little girls and boys skate around on it... I get its a little game.


What I don't get is why its on the google home page, usually when they do something like this it makes sense, its a reference to some historical event or anniversary... This one seems to be celebrating, with no explanation, the fact that its cold outside and we've had a bit of snow.



And you know what, I think I know where it came from.  I just got through re-reading "I'm Feeling Lucky" one of the few candid retrospectives about the internal workings and history of Google.  And the author, one Doug Edwards, goes into pains to explain just this kind of thing going on... Random changes to the branded headed title page... He was frustrated by it... Today, so am I.

Tuesday 15 January 2013

Trusting Code

I like to write self contained prototype code, when I'm mocking up a system.  And usually I clean this code up from prototype to acceptable to production quality as a project progresses, I accept sometimes that there's not enough time during a project to smarten everything up, or where functionality comes before form, but generally one wants to avoid leaving potentially buggy, unaired, unoptimized code hanging around too long.


So, its with amazement that I see in the latest version (supposedly brand new from the ground up) there's a project rewrite still containing code I wrote, for a different project, two years ago.

The item in question is a GUI control, which explains its longevity, but the new developer has not checked this code in use, they've in fact taken it verbatim that the code is fine just because its there to be used.

I hate this.

I hate even picking up other libraries from online sources without carfully going through the code, which is why I generally start a project researching to work out absolutely which parts of this wheel I need to reinvent this time, and which I might trust.  I don't automatically give my trust to other people's code.

But, it seems many other developers do... Folks, don't.

French to Scotch

Do you think they translated that French fella into a Scottich accent cus of his goatie?

Saturday 12 January 2013

Max

Just sharing a picture of one of our dogs, Max... He's a silver toy poodle and we're putting him out there in the world as a stud dog in 2013.


Or to give him his full name Maximus Myridius Didious...


By the way, the yellow golden eyes, that's just the flash, he has lovely dark brown eyes.

Asus P6X58D-E Failed

I've checked my machine out now, and I've come to the conclusion that all the peripherals in this machine work fine else where its down to the Motherboard.

The machine will boot, one time in one hundred, so the CPU is working it lets me boot into a live Linux CD and even boot Vista, if it boots, and its during these other ninety nine attempts to boot that the problems come along.

The problem is this, turn the power on, hit the power button (on the motherboard or on the case) and the PSU will spring to life the CPU Cooler and the Graphics card fan, all these fans spin at maximum speed, and that's it...

No BIOS, no POST, nothing... Even with a speaker attached, nothing, no beeps, nothing.

Now, this usually indicates a dead PSU, but I've tried the it in another of my machines, and its fine!

Even if I remove all the RAM, which is meant to give long beeps in sequence, the motherboard doesn't squeek, nothing, it just sits there and blank black max fan time.

The problem I have now is I need to get service on it, this thing cost me £143 on December 2010.  That's two years one month.  It gets from Asus a 3 years warranty, so I should be covered, Asus however have told me to get hold of the retailer, the retailer have yet to reply - I bet they want me to use their 60p a second help line and sit on hold for an hour.

The rub though is, even if they do try to help, or I get an RMA... my CPU is a socket 1366, its not going to just randomly fit another motherboard and 1366 boards are getting very rare on the market (for reasons which I've had no interest in, but am now trying to fathom out).  I have literally just sent missives to the retailer for assistance, I can at least prove my warranty started on the 11th November 2010, and hence is within the mandated 3 years by Asus.

And there's another problem, Asus state a 3 year warranty, but they say to contact the retailer about it?

I'll let you know how I get along.

Galant Ikea Desk
The other battle I'm fighting is about my desk, I lowered my desk last night, but one of the legs felt dead loose, you can unscrew the legs and they let you telescope them in and out, well one of them just feels all floppy and will not tighten back up.

I'm a little worried about this, but tomorrow sometime I hope to get all the crap off of the desk and to put all the legs right and level.

Friday 11 January 2013

PC Out

My main PC at home has gone tits up, this is a pretty technical problem, the thing won't even post, it doesn't even beep a warning sequence from the motherboard.

I've checked and rechecked everything all to no avail.

Tonight is going to be a complete machine tear down and check, whatever is wrong... It has me worried.

Tuesday 8 January 2013

Chairs, Cars and my Annoyances


Today's a bit of a birc-a-brac post for the world at large, firstly I'm sick of the high street media were going on about this just this morning, about how internet shopping is killing shops and shopping in person.  Well, let me tell you, it's not only the internet its stupid annoying shop staff.

Let me run you through this properly, I need a new chair for my office, as a programmer I spend a lot of time in my chair, but with lots of expenses over Christmas I had a set budget to adhere to for a new chair, we're talking less then £100.  So, with such a budget I wanted to avoid getting a £39.99 plastic armed monster and have it fall to bits within a year, I actually wanted pretty much the same chair I had from 2002 til 2008, a leather short backed operators chair.

I have a similar chair at work, but its price is prohibitive, however, it looked like a certain store had the answer, a chair very much what I wanted, for £73.  So I set off to try it out last night.

I got there, and I tried it out, but was confused to see the price was £110.  Beyond the budget, and not as stated on their website, so I asked for a price check... I was made to feel like an utter and total leper.  We're not talking about a performance car show room, where if you ask the price you're instantly known to not be able to afford the vehicle, we're talking about a poxy chair in an office furniture shop.  I'm stood there waiting to speak to this chap, we'll call him 'chap #1', I'm stood there and behind him are 'chap #2' and 'chap #3', they ignore me.

I speak to 'chap #1' ask for a price check, he comes with me looks at the item, walks back to his post and says to the second guy "Can you price check that chair".  Now, this is where I got angry, the other guy walks the same path up the store to the same chair and walks back and then gets out his phone, he doesn't say "wait a moment please" or acknowledge me in anyway, he just says to 'chap #1' "back to your post".  This guy is ignoring me, his name tag denotes him as the "Store Manager"... He's ignored me and making me feel awkward, as rather than being able to engage with him I'm just stood there, staring and fiddling with his phone... he is going to use the company website to check the price... he can't seem to make any managerial decision himself.

This manager fiddles and piddles about and finally says... "Price is right" and goes to walk off.  Literally, he just turns his eyes.... "Price is right" and turns back to his minion in the form of Chap #3 to carry on chatting with him... Customer service on par with making me feel like a pile of shit.

So I interject, "but the price is £73".

"Ah no" chap #2 says with smug satisfaction "that's the price if you buy two, see" and he shows me the phone, which says "From £73.33", they're not from that price at all folks, they're from £110. This miss-leading advertising annoyed me, being left just stood there annoyed me, so folks stay out of Staples at Lady Bay Retail Park in Nottingham, they're not much cop.

And now my other piece of bric-a-brac (http://www.bbc.co.uk/news/uk-england-sussex-20944339) this poor woman, I know now that her insurers will either not pay out for her, and even if they do they're going to say she claimed, loosing her no-claims, she'd be all right if she just rolled the car forward and touched bumpers with the vehicle the neck patients were in, then she could claim off their insurance, but as it stands I think this poor woman is going to get utterly screwed over.  And this just emphasises why I hate Insurance companies.

Monday 7 January 2013

Python - Warning Lessons


Over the last fortnight I've been dabbling with Python, the seemingly ubiquitous language seeping into every nook and cranny not containing a dedicated C++ follower, like myself.

Its on the Pi, its behind a lot of research work and big companies, like Google (before they worked on Go - lots of their stuff was Python, and perhaps still is).

So, I took a look... Its a powerful language, I don't doubt, it reminds me very much of writing procedural Pascal form the mid 1990's, but most importantly it looks like psuedo code, other authors mention this (http://www.jayconrod.com/posts/37/a-simple-interpreter-from-scratch-in-python-part-1) and others even crow about the syntax and structure being so simple.  But I hate it.

It leaves so many stones unturned, so many open ends, and I have to say it makes me cringe.

In C++ one has to definiately define the article of the body of a class, for example:

class Alpha
{
private:
int m_Value;
};

Now, the value of Alpha can only be set by functions which are part of this class, it is totally private:

class Alpha
{
private:
int m_Value;
public:
Alpha (const int& p_Value)
: m_Value (p_Value)
{
}
};

We can all agree this is a powerful tool, it lets us change and range and control the use of variables in the class, it can make our code safe and remove bugs, also the type of m_Value is implicitly defined one can't define m_Value and then start to point values at m_value by mistake, the compiler tells you to go away, this in my opinion is correct programming, I agree there's a lot of overhead (if you let the compiler create all that code) but there are far less potential bugs.

In Python we can get the same class:

class Alpha:
    def __init__(self,p_Value):
        self.m_Value = p_Value

And I find this abhorrently dangerous, first off, I didn't have to create "m_Value" the interpreter simply saw the name and invented it, if I then use "m_value" or any typo the interpreter does not thing, Ah hang on I don't know what this value is, it thinks "Ah a new thing to work with" and creates said typo as a new value.  Debugging such things is an utter nightmare.

Then the syntax itself, its clean, though not neat the reliance on tabbing to define structure annoys me.  I already have a big bee in my bonnet about different indentation styles in the C language family, so to see indentation made a part of the language, white space defining function, is scarey as hell.

But, Python is out there, and its a language I don't know an aweful lot about, so I'm learning it, gradually on the side.  Just you guys and gals out there learning it be very wary of its pit falls!

Thursday 3 January 2013

HTML5 SSE Chat Server Example (Full Code & Video)


My first post of the year, and I should say Happy New Year I suppose.  The year is already starting out better for you poor folks reading my non-sense blog, because I have an item I'm going to explain in detail for you, including videos and pictures, and this is how to set up a very simple anonymous chat system through a server to multiple browsers using HTML5's Server Side Eventing mechanisms to remove the need to write polling javascript code in your browser.

The reason I'm going to cover this in details is that though there are lots of examples and descriptions out there as to how to set about doing just this the example code given is sketchy in most cases, missing in many and does not actually accomplish much with the technology to plant the seed of actually putting it to use.  So, in the hope of setting a trend for the year I'm going to put this post together, take video and explain it all and hopefully you'll all be able to get to the same stage I'm at and have a working chat exchange program.

Prerequisites
Lets cover the prerequisites, you need a web server set up on a machine, in my example here I'm going to use apache2 on Ubuntu 12.04 LTS, with PHP5 as my server side scripting language.  The browser I'm going to use is Google's Chromium, because I like its debugging in the form of the javascript console.

The server I have has been set up as a blank server image on a VMWare virtual machine instance, and I've run the following commands to get it set up:

sudo apt-get install apache2 php5 libapache2-mod-php5
sudo /etc/init.d/apache2 restart
sudo apt-get install apache2 php5-mysql mysql-server
sudo /etc/init.d/apache2 restart
sudo apt-get install phpmyadmin

I allowed phpmyadmin to install and selected to install it into "apache2" at the prompt and I'm done, my passwords are all known to me and so whenever I've been asked for one I've typed it in, you will need do the same, and no, I don't want to know your passwords :)

I've gotten the server's IP (ifconfig) and from my desktop machine I've browses to its IP "http://192.168.2.101/phpmyadmin" and checked I can get onto the database.

Database
I'm using a database and a single table as an imtermediary, each browser connected to the chat page will insert lines of text "chat strings" and they will be inserted in time order into the table and then delivered to all the listening clients as they cycle around.

This processing of the chat potentially allows us to process and filter it for profanity, or prevent inappropriate discussions of any type we choose, before we insert it into the database.

We need to add a database with a table to the server, and so we're going to use this code:


(Don't worry this is an image, we're going to break down the SQL line by line).

CREATE DATBASE ChatSystem CHARACTER SET latin1 COLLATE latin1_general_ci;

This line actually creates the database for us, its called "ChatSystem" and uses the latin character set and sorts (collates) its columns in tables by index against the latin character set.  If you're intenting to use a different character set you'll have to hit your chose SQL server's documentation for their available character sets.  But latin1 is going to cover most of Western Europe, North and South America's and the Antipodes for this example.

USE ChatSystem;

This just selects the database we just created.

CREATE TABLE IF NOT EXISTS GeneralChat
(
    Id   BIGINT NOT NULL AUTO_INCREMENT,
    PRIMARY KEY (Id),

    Tme     BIGINT NOT NULL,

    Text     VARCHAR(128)
)
Engine = InnoDb;

This instruction does pretty much as you would guess reading it, it creates the table with the three fields "Id", "Tme" and "Text".  Text is just the chat string we're exchanging, the field design allows for upto 128 characters so in our HTML presented to take chat strings in we must remember to limit the input box's length to allow only 128 or less characters.

Tme is actually time, I'm omitting the 'i' to avoid using a reserved word in anyones SQL implementation, the time we're going to take as the time the server receives the chat string.  The client browser is not going to have to send it, and its just going to be a number becuase our server side scripting language (PHP) gives us the command "time()" which returns the time as a single integer.

Finally Id is the order the chat strings are inserted, we're going to leave this as an auto incremented integer because then we need not worry about it in our SQL insert statement later, again like Tme the Id need not be worried about nor generated by the client browser.

I take all this SQL and run it through phpmyadmin's SQL intepreter and I get my new database and table ready for use.

Client Side - Chat.html
On the client I'm going to have a very simple page, which is an outer page with an input box and a "send" button, and then an inner frame (iframe) which is showing the messages as they come back from the server... I could make this all one page, but I'm splitting things up to keep the demonstration as simple as possible.

The outer page first, we need to have a table (yes I like tables you can use divs and spans all you like) which has two rows, the top row is going to show us chat arriving and contain our iframe and below we're going to have our input text box:

<table>
<tr>
<td>
<iframe src="chatstream.html"></iframe>
</td>
</tr>

<tr>
<td>
<table width="100%" border=0 cellspacing=0 cellpadding=0>
<col width="85%">
<col width="15%">

<tr>
<td colspan=1>
<input id="ChatString"
type="text"
value=""
maxlength="48"
onkeypress="HandleKeyPress(event)">
</td>
<td colspan=1
onclick="UserSubmitChat()">
Send
</td>
</table>
</td>
</tr>
</table>

So we have several things we need now define, first the inner frame HTML "chatstream.html", which we'll cover in detail below.  And then two javascript functions:

"HandleKeypress(event)" and "UserSubmitChat()", the first function just checks whether we're handling the enter key and calls the second function, it looks like this:

function HandleKeyPress (event)
{
var l_chCode = ('charCode' in event) ? event.charCode : event.keyCode;
if ( l_chCode == 13 )
{
UserSubmitChat();
}
}

So the mor importand function is "UserSubmitChat()" which looks like this:

function UserSubmitChat ()
{
var l_textBox = document.getElementById ("ChatString");
if ( l_textBox )
{
InsertChatString(l_textBox.value);
l_textBox.value = "";
}
}

Very simply the function looks up the element called "ChatString" which is our input box (see above in the HTML) and if found it inserts the chat string, blanking the input box.

Note: One could optimise the page to take the "document.getElementById" call once and store the "l_textBox" throughout!

So, I'm simply taking the string content of the input box and calling yet another function, what is this "InsertChatString(value)" function doing?  Well, lets take a look:

function InsertChatString (p_ChatString)
{
m_ChatQueue.push(p_ChatString);
}

Well, it takes the string and pushes it onto the queue of strings going somewhere, this is important we want to order the chat we're posting and a queue is an easy way to achieve this, we push onto the back of the queue and pop off the front when we want to dispatch the string somewhere.

Few developers realise the queue functionality is built directly into the javascript array, so to allow the member "m_ChatQueue" to be a queue for us we simply have to create it as an empty array:

var m_ChatQueue = [];

So, all in all our javascript block now looks like this, with neat and tidy functions to handle key pressing, string extraction and pushing the string onto the queue.

<script language="javascript">
var m_ChatQueue = [];

function InsertChatString (p_ChatString)
{
m_ChatQueue.push(p_ChatString);
}

function UserSubmitChat ()
{
var l_textBox = document.getElementById ("ChatString");
if ( l_textBox )
{
InsertChatString(l_textBox.value);
l_textBox.value = "";
}
}

function HandleKeyPress (event)
{
var l_chCode = ('charCode' in event) ? event.charCode : event.keyCode;
if ( l_chCode == 13 )
{
UserSubmitChat();
}
}
</script>

But still the strings go no-where, what I have done is therefore rather than have an iframe or hidden on screen element posting (with an HTML form) the data I've decided to have a server side script page (php) which will take the chat string as a HTTP GET parameter.

This page is to be called "chatsubmission.php" and take the paramter "ChatString", so the URL for posting the chat string "HelloWorld" will be "chatsubmission.php?ChatString=HelloWorld".

When I open the chat posting page then I want to smooth out the posting of these strings to the server, as the user enters chat strings I don't want them to be paused out as it posts to the server I want things to flow.  So the user seamlessly queues the chat strings they enter, and they may have posted many before the page submits them all, but the submission goes on as a background task very fast without seeming to interrupt the user or their browser.

To achieve this I'm going to use a timer, and when the HTML loads I'll start it:

<script language="javascript">
var m_ChatTimer = null;

function StartChatTimer ()
{
StopChatTimer ();
m_ChatTimer = setInterval(ChatTimerMatured, 500);
}

function StopChatTimer()
{
if ( m_ChatTimer != null )
{
clearInterval(m_ChatTimer);
m_ChatTimer = null;
}
}
</script>

Note: All the functions shown all go in the same single "<script>" block on the same page, I'm splitting them up for this post only, and will post the whole page code at the bottom.

The HTML to start the timer is then:

<body onload="StartChatTimer()">

Now we have a timer maturing every 0.5 seconds, and its going to submit to some URL we we've defined all the strings of text in the m_ChatQueue.  What does this code look like?

function ChatTimerMatured ()
{
StopChatTimer ();

if ( m_ChatQueue.length > 0 )
{
while (m_ChatQueue.length > 0)
{
var l_value = m_ChatQueue.pop();

var l_URL = "chatsubmission.php?ChatString=" + l_value;
var l_HTTP = new XMLHttpRequest();
l_HTTP.open ("GET", l_URL, false);
l_HTTP.send (null);
}
}

StartChatTimer();
}

Let us just analyse this code, so we stop the chat timer, to prevent another timer mature happening whilst we're processing a potentially long list of strings of chat.  And then check the queue, if there are queued items we take one item off (pop) at a time and build the submission URL.

Armed with the URL, we then cheat, we use the javascript XMLHttpRequest object to get that URL, effectively then our server side page gets the request and parameters.  Much as if we had an HTML form and submitted it causing an HTTP Post.

Take a moment to look this code over, both above and in the full listing for "chat.html" at the bottom of the post.

Once all the strings are posted, or we're determined there's nothing to do, we start the chat timer again.  During this interlude the user may have posted more chat strings, but we smooth out the operation.  We could also temper the duration of the timer to better suit slow internet connections, or increase the responsiveness if necessary.

Server Side - ChatSubmission.php
So far we've taken the user input and cleverly posted it into the server, now we must see what the server side does with the chat string:

<?php

if ( !empty($_GET) )
{
require_once('dbFunctions.inc');

$l_ChatString = $_GET["ChatString"];
if ( InsertChat($l_ChatString) )
{
echo "Y";
}
else
{
echo "N";
}
}

?>

This is the complete "ChatSubmission.php" page called by the javascript XMLHttpRequest object we previously explored, the PHP page includes some functions (written by us) and then from the URL gets the "ChatString" parameter, if there is such a parameter it then calls the function "InsertChat".

The PHP function I have written for this looks like this:

function InsertChat ($p_ChatString)
{
$l_result = false;
$l_Connection = ConnectDatabase();
if ( $l_Connection != null )
{
$l_ServerTime = time();

$l_SQL = "INSERT INTO GeneralChat VALUES (null, $l_ServerTime, '" . $p_ChatString . "')";

mysql_query($l_SQL, $l_Connection);

CloseDatabase($l_Connection);

$l_result = true;
}

return $l_result;
}

It opens the database connection, builds the insert SQL and performs the insert, returning true upon success or otherwise false.  Very simple, and you may pick over the bones in the full listing.

Before we move on, we need to just see the "$l_ServerTime" and use of "null" in the SQL we insert with, the null is for the Id, which as you should recall we set as AUTO INCREMENT, and the time is the server time at which we receive the message.  This time will be of interest later.

So, there we have the first leg of our chat string's journey, from the text input box into a javascript queue and then on a timer maturation out to the server as a seamless posting and into the mysql database.

Note: You can use any database or server side language you want at this point!  Change the script around however you want and experiment!

Server Side Event - Second Leg
The second leg of our chat strings journey is to be from the database out to the screen, this is performed with the "chatstream.html" page, which is going to set up the client browser to listen for the server side events being pushed back.  This is the page which you'll see most demonstrated out there on the internet, most demo's cover this setting yp of the message reception, and many do it better than I.

There are basically three events you can handle with Server Side Events in HTML5, "open" which is the event of the connection to the server being established, "error" which is the event when something has gone wrong server side, and finally "message" which is a text based string of characters from the server.

What you choose to put into the event is up to you, there are examples of doing json, xml or just plain text.  For our chat system the message sent is going to be the actual raw chat string.  We're going to make sure the server sends one chat string per message.

So, lets take a look at our HTML, its very simple:

<html>

<head>

<script language="javascript">
...
</script>

</head>

<body onload="LoadPage()">

<table id="chatTable" border=1 width="100%">
<tr>
<td>Chat</td>
</tr>
</table>

</body>

</html>

So we have a table called "chatTable" and we call a function called "LoadPage" when we open the page in the browser, we can now define the javascript:

function LoadPage ()
{
m_ChatTable = document.getElementById("chatTable");

source = new EventSource('chatlist.php');
source.addEventListener ("message", ChatArrived, false);
}

Breaking this down, we grap the chatTable from the HTML and then set up the key SSE receiving object.  This "EventSource" points to a new server side php page (chatlist.php).

And then registers for the "message" arriving to point to javascript function called "ChatArrived":

function ChatArrived (event)
{
console.log(event.data);
}

For debugging, this function will suffice, but my full function takes the data string and appends it to the chatTable:

<script language="javascript">
var source = null;
var m_LastTime = 0;
var m_ChatTable = null;

function LoadPage ()
{
m_ChatTable = document.getElementById("chatTable");

source = new EventSource('chatlist.php');
source.addEventListener ("message", ChatArrived, false);
}

function InsertChat (p_ChatString)
{
if ( m_ChatTable != null )
{
var l_newRow = m_ChatTable.insertRow(m_ChatTable.rows.length);
var l_newCell = l_newRow.insertCell(0);
l_newCell.innerHTML = p_ChatString;
}
}

function ChatArrived (event)
{
InsertChat (event.data);
}
</script>

This therefore completes the HTML5 SSE loop back, so long as the "chatlist.php" serves up SSE events each time it sends a message it'll arrive on the page and append to our chat table.

Note: Some readers have forgotten this "chatstream.html" page we've just defined is inside the iframe from the first page, so we can see on the table the chat listing in the iframe and then the input box below.

Server Side - chatlist.php 
Our final page is the PHP page which lists the chat back to the clients, the design I've gone with is a loop which takes the time and then returns each line of chat between the time taken and "now" each loop.  Then it sleeps and repeats, it looks like this:

<?php

header("Content-Type: text/event-stream\n\n");
header("Cache-Control: no-cache");

require_once('dbFunctions.inc');

$l_LastTime = 0;
while ( true )
{
$l_LastTime = ListChat($l_LastTime);
sleep(1);
}
?>

The important parts for the HTML5 SSE mechanism are the two header lines, we define the header type to be "text/event-stream" and that the page is to use no cache on the client side.

Note: The two new lines "\n\n" on the Content-Type string are very important and should not be ommitted.

Again the PHP page uses the include file we've defined, which has its complete listing below, but we'll just cover the "ListChat" function now:

function ListChat ($p_LastTime)
{
$l_LastTime = $p_LastTime;

$l_Connection = ConnectDatabase();

if ( $l_Connection != null )
{
if ( $l_LastTime == 0 )
{
$l_LastTime = time();
}
$l_Time = time();

$l_sql = "SELECT * FROM GeneralChat WHERE Tme > " . $l_LastTime . " AND Tme <= " . $l_Time;

$l_result = mysql_query($l_sql, $l_Connection);
if ( $l_result )
{
$l_numRows = mysql_numrows($l_result);
if ( $l_numRows > 0 )
{
for ($l_i = 0; $l_i < $l_numRows; ++$l_i)
{
$l_LastTime = mysql_result($l_result, $l_i, "Tme");
$l_Message = mysql_result($l_result, $l_i, "Text");

sendMsg ($l_LastTIme, $l_Message);
}
}
}
CloseDatabase($l_Connection);
}

return $l_LastTime;
}

Important to the operation of this function, it takes the last time we checked for chat and returns the "now" time for the last message dispatched.  This lets us then query in the center for only those lines of chat we need to send.

The flow of the function simply opens the database, builds the SQL, performs the query and if there are results dispatches each row of results (each chat string) as an individual event to the client side.  Our client handles this in its "ChatArrived" function and posts each row onto the onscreen table.

We also specially make sure if the time of the last query was zero (i.e. the script just started running) we correct to "now" to prevent the whole history of the text being stremed back to the client.

The key piece of code however is this snippet:

for ($l_i = 0; $l_i < $l_numRows; ++$l_i)
{
$l_LastTime = mysql_result($l_result, $l_i, "Tme");
$l_Message = mysql_result($l_result, $l_i, "Text");

sendMsg ($l_LastTIme, $l_Message);
}

Each result takes its time, which we use as a value to send the string against, and the string, the sendMsg function then looks like this:

function sendMsg($id, $msg) 
{
echo "id: $id" . PHP_EOL;
echo "data: $msg" . PHP_EOL;
echo PHP_EOL;
ob_flush();
flush();
}

Again, many of the online examples of SSE give this style of function as their example, we add the Id to the PHP echo (which is going back down the text/event-stream to the client) and the "data: " line is the line which is passed to the "ChatArrived" function as the "event" parameter.

So if we put the code:

echo "id: 1" . PHP_EOL;
echo "data: HelloWorld" . PHP_EOL;
echo PHP_EOL;
ob_flush();
flush();

Then the "ChatArrived" function would receive a parameter of "HelloWorld", not "data: " leading it!

And this folks is the whole example.... You can follow the flow now:


  • chat.html : Takes the input string, queues and then dispatches the strings
  • chatsubmission.php : Accepts the dispatched strings into the database
  • chatstream.html : Sets up the SSE event source and registers to handle messages from chatlist.php
  • chatlist.php : Sends the messages as they arrive into the database

What follows is a live stream of my performing all these steps with a Ubuntu laptop and a Ubuntu server (Recorded 1st Jan 2013):



/// chat.sql full listing ----------------------------------

  CREATE DATABASE ChatSystem
  CHARACTER SET latin1
COLLATE latin1_general_ci;

  USE ChatSystem;

  CREATE TABLE IF NOT EXISTS GeneralChat
  (
Id BIGINT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (Id),

Tme BIGINT NOT NULL,
Text VARCHAR(128)
  ) Engine = InnoDb;


/// chat.html full listing --------------------------------

<html>

<head>
<title>Chat</title>
<script language="javascript">
var m_ChatQueue = [];
var m_ChatTimer = null;

function InsertChatString (p_ChatString)
{
m_ChatQueue.push(p_ChatString);
}

function UserSubmitChat ()
{
var l_textBox = document.getElementById ("ChatString");
if ( l_textBox )
{
InsertChatString(l_textBox.value);
l_textBox.value = "";
}
}

function HandleKeyPress (event)
{
var l_chCode = ('charCode' in event) ? event.charCode : event.keyCode;
if ( l_chCode == 13 )
{
UserSubmitChat();
}
}

function StartChatTimer ()
{
StopChatTimer ();
m_ChatTimer = setInterval(ChatTimerMatured, 500);
}

function StopChatTimer()
{
if ( m_ChatTimer != null )
{
clearInterval(m_ChatTimer);
m_ChatTimer = null;
}
}

function ChatTimerMatured ()
{
StopChatTimer ();

if ( m_ChatQueue.length > 0 )
{
while (m_ChatQueue.length > 0)
{
var l_value = m_ChatQueue.pop();

var l_URL = "chatsubmission.php?ChatString=" + l_value;
var l_HTTP = new XMLHttpRequest();
l_HTTP.open ("GET", l_URL, false);
l_HTTP.send (null);
}
}

StartChatTimer();
}

</script>

<body onload="StartChatTimer()">

<table>
<tr>
<td>
<iframe src="chatstream.html"></iframe>
</td>
</tr>

<tr>
<td>
<table width="100%" border=0 cellspacing=0 cellpadding=0>
<col width="85%">
<col width="15%">

<tr>
<td colspan=1>
<input id="ChatString"
type="text"
value=""
maxlength="48"
onkeypress="HandleKeyPress(event)">
</td>
<td colspan=1
onclick="UserSubmitChat()">
Send
</td>
</table>
</td>
</tr>
</table>

</body>

</html>

/// dbcredentials.inc -- Full Listing ----------------------

<?

$g_User = "root";
$g_Password = "********";   // Place your password here
$g_Database = "ChatSystem";

?>

/// dbfunctions.inc -- Full Listing -------------------------

<?

function ConnectDatabase ()
{
$l_Connection = null;
include ('dbCredentials.inc');

$l_Connection = mysql_connect ('localhost', $g_User, $g_Password);
if ( $l_Connection )
{
if ( !mysql_select_db($g_Database) )
{
mysql_close($l_Connection);
$l_Connection = null;
}
}

return $l_Connection;
}

function CloseDatabase ($p_Connection)
{
if ( $p_Connection != null )
{
mysql_close($p_Connection);
$p_Connection = null;
}
}

function InsertChat ($p_ChatString)
{
$l_result = false;
$l_Connection = ConnectDatabase();
if ( $l_Connection != null )
{
$l_ServerTime = time();

$l_SQL = "INSERT INTO GeneralChat VALUES (null, $l_ServerTime, '" . $p_ChatString . "')";

mysql_query($l_SQL, $l_Connection);

CloseDatabase($l_Connection);

$l_result = true;
}

return $l_result;
}

function sendMsg($id, $msg) 
{
echo "id: $id" . PHP_EOL;
echo "data: $msg" . PHP_EOL;
echo PHP_EOL;
ob_flush();
flush();
}

function ListChat ($p_LastTime)
{
$l_LastTime = $p_LastTime;

$l_Connection = ConnectDatabase();

if ( $l_Connection != null )
{
if ( $l_LastTime == 0 )
{
$l_LastTime = time();
}
$l_Time = time();

$l_sql = "SELECT * FROM GeneralChat WHERE Tme > " . $l_LastTime . " AND Tme <= " . $l_Time;

$l_result = mysql_query($l_sql, $l_Connection);
if ( $l_result )
{
$l_numRows = mysql_numrows($l_result);
if ( $l_numRows > 0 )
{
for ($l_i = 0; $l_i < $l_numRows; ++$l_i)
{
$l_LastTime = mysql_result($l_result, $l_i, "Tme");
$l_Message = mysql_result($l_result, $l_i, "Text");

sendMsg ($l_LastTIme, $l_Message);
}
}
}
CloseDatabase($l_Connection);
}

return $l_LastTime;
}

?>

/// ChatStream.html --- Full Listing ------------------------

<html>

<head>

<script language="javascript">
var source = null;
var m_LastTime = 0;
var m_ChatTable = null;

function LoadPage ()
{
m_ChatTable = document.getElementById("chatTable");

source = new EventSource('chatlist.php');
source.addEventListener ("message", ChatArrived, false);
}

function InsertChat (p_ChatString)
{
if ( m_ChatTable != null )
{
var l_newRow = m_ChatTable.insertRow(m_ChatTable.rows.length);
var l_newCell = l_newRow.insertCell(0);
l_newCell.innerHTML = p_ChatString;
}
}

function ProcessChat (p_ChatString)
{
InsertChat (p_ChatString);
}

function ChatArrived (event)
{
ProcessChat (event.data);
}
</script>

</head>

<body onload="LoadPage()">

<table id="chatTable" border=1 width="100%">
<tr>
<td>Chat</td>
</tr>
</table>

</body>

</html>