Virgin Media : Poor Internet Speed

I'm continuing to have my issues with Virgin Media, let me explain.  Last month I came to them asking to remove the superfluous junk offering items, as I was simply paying more for items I didn't use, want nor need.

They could not help, in fact they flat refused to, so I cancelled 66% of my services off completely, TV, Phone all gone and never ever to return - well done them...

The next item was the 200mbit broadband I was paying for, below is a historic speed test I carried out a lot, as you can see I never, ever, got near 200Mbit, but I always got over 50Mbit...

Being quite a logical person therefore, and based on this evidence, this proof I had for myself, I downgraded the internet from the Vivid200 to Vivid50, i.e. 50mbit... Yeah, save money, it can do 50 mbit easy, right... Right... RIGHT?.....

WRONG.

This is tonight's speed test...

Last night it was slower than that, I had a download speed of 1.22mbit, and upload of 620K.

Yeah, rocking the 1990's internet speed.

And I'm utterly sick of it, I'm sick of paying for a service I never ever get.  It may have been optimistic, maybe even naive to think having seen 50mbit when paying for 200, I would get 50 when paying for 50, but clearly something has gone astray here.

Clearly, Virgin Media are up to some shenanigans, be that throttling, load balancing, whatever term they want to give it, or even what ever they want to say in denial, but I'm observing this poor speed, I'm enduring this poor service, and I'm utterly and totally frustrated with the script (I would say expert system, but it's clearly a script) the people in the call centers follow.

This is NOT my equipment, this is not my network, this is their external speed, this is from their superhub 2 and outwards, NOT the internal side of that connection.

That is a test through the newer version of Ookla (the site at the top) as you can see the speed is even worse twenty minutes on - BTW, note to Virgin Media, your own engineers use this site to do a test upon installation.

And here's the speed an hour after my original post... this is now 21:32 on a Sunday evening...

Server Admin : Ubuntu 17.04 thinks it's Ubuntu 12.04???

Yeah, I'm serious, I've taken time tonight to look at the release of Ubuntu Server 17.04, specifically to set up a new mini-server which is to be Core 2 Duo powered and on 24/7 as boot strapper & service strapping server itself.

But, before I run I like to walk, so I set up a 2 core, 1GB RAM VMware machine from the 17.04 ISO... Take a look at the first thing it has presented to me....

Yes, just read that again... I booted the server... and the only action I took was to log in... Welcome to 17.04... All good...

What wait?.. Why am I being warned to upgrade my 12.04?  This is 17.04?

Before I ran around like my last vestiges of hair were on fire, I decided to do a simple test, I've previously found that Ubuntu often goes wondering off on the internet for message of the day (motd) information, so I pulled the network card (virtual) out of the machine.

This results in a long boot time, but you at least know no remote files or services are going to be listing things on your screen...

Five minutes later, I get to see what the system says... From experience I think Canonical have shown some news and up to 80 characters of 40 lines of news, I've never seen that much, but it has been a while since I looked at their motd scripts.

After logging in, I still got the message, however, a fresh install (without any network) didn't show the message, so I believe the install we see here cached something from an online source.

Anyway, taking a look in the /etc/update-motd.d folder, you can see a series of numbered scripts, these are so numbered to allow Canonical, or yourself, to add message of the day scripts, and keep them in the order you see them.

Checking "00-header" we see just the usual log in.

Then "10-help-text" is the three lines about documents, management and support.  I actually add "#" to the start of each of those lines to remove that files actions, I don't delete the file though, just in case.

The next line "50-motd-news", this looks to be the culprit... I'm not even going to look inside the file, because I can see the next file in the folder is "90-updates-available" and I can see in the login that the updates available happen after the message I want to be rid of....

So this strange, confusing, message is in "50-motd-news", I'm going to cut to the chase and kill that file.

And now my login is much neater, I have added a call to "ifconfig" into the 10-help-text, but my login is now clean of this strange message.  But I'm not impressed this has gone on, and I'm going to have to take a look through all these other motd scripts to see what and where my server is going off to.... Hmmm.

Sys-Admin/Dev Ops : Assumption is Danger

As a systems admin, or dev ops, or whatever your job title might be, never ever assume that the person you're handing a system to has a clue.  This might seem harsh, but it's true, and proves itself true time and time again.

"Assumption is the mother of all f**k ups"

About a year ago I deployed a system which automatically sent requests to remote machines (via SMS) getting those machines to report their status or send back error information, but also to gather some basic information.

It has run happily for a whole year, it has been all pretty plain sailing, the hours and hours of work I put into it, to automate it and keep it self-sustained have paid off, zero faults, zero down time, self-regulation is the way forward for me; even if it took slightly longer to put the system in place, it has needed no human input for nearing a year!

However, the unit needed to move, about a week ago, it needed be physically picked up and taken out of my small server room and into the official server room, a dark cupboard basically controlled not by myself or my cohort, but the IT boffins.

Fine, I notified the customers, went off to the IT area, sorted out who I was to hand it to and physically delivered it to the chap, I watched him start to plug it all back together, power, wires, boot, fine....

I assumed he'd do this seamlessly....

Until this morning, well a morning last week, as I post these with a date in the future.  That morning was hell, I walked into a wall of customers not being able to get to their machines, the Easter weekend was looming, performance needed to be monitored, customer sites didn't have regular staff, explaining to temporary cover staff that system would be off was not a prospect I relished. 

To be frank, a lot of flapping going on, more than I expected... IT reported the system back online, but customers didn't stop flapping... Indeed, none of the estate seemed to be able to connect in... 1 hour, 2 hours, I've asked the boffins to check it time and again "It's fine", they tell me.

I look locally, I can't see the controller machine on the network, I can't see it through the remote management console... Where the hell is the machine?

I assure the customers I'll have answers within the hour, I hit social media with the same, this is going very public, and I'm rather annoyed as for a whole year things have run seamlessly; but been ignored, now its offline for a scheduled purpose and everyone is complaining, I do not want my success wiping away in a flood of negative press.

I call the IT boffins... "we'll look into it"... No, no no, you'll get onto it right now, not look, not glance, answers are needed.  Action from you is needed before my Re-Action goes nuclear.

I wait, five minutes, I was willing to give them ten.... My phone rings...

Them > "Hello?"...

Me > "Answers?"...

Them > "Yeah, you know when you brought it back?"...

Me > "The Machine?"....

Them > "Yes"...

Me > "I remember, why?"...

Them > "Well, it has power"...

Me > "Good"....

Them > "Not really"...

Me > "Why not?"...

Them > "Because that's all it has, it's not been plugged into the network"

I hung up.  They plugged it into the network, I had a slew of data come through... The customers were pacified.

I however was not.

I've had an on the spot review, firstly the IT bod who did this was held to account, second I was held to account for not noticing.

In not noticing I admit that having had it run cleanly for a year I had turned off the performance reports and I admitted I had assumed a network machine being handed to an IT bod would be plugged into the network.  People were not happy, least of all me, but that was the fall out.

However, I then had to do a tertiary clean up and after the Easter break I spoke to three of my main customers, trusted operators, the actual folk who should have been using the machines at the remote sites; not temporary staff; I asked them why they had not noticed.  The replies...  "Because it had worked for so long without an issue", "like you make it work, so we just guess it always is" and "we didn't notice it was offline".

They were very much putting everything into my court, assumption on the part of all parties was to blame.

The lessons learned for me are to now keep checking, keep monitoring, use my automation to report status, to fix faults and if human errors creep in, to let me know.

I'm now off to spec up a service I can run on one of my own servers, just to ping the network machine which went AWOL and receive a report from it to let me know what its up to, this might be a bit of python or just bash on a cron task, but it's going to be something rather than nothing.

I will NOT assume again.

Server Admin : How Good is your Backup?

How robust is your back up solution?  Go on, be honest with yourself, how good is it?... Because I've seen a whole host of them and, at this very moment, this is the screen up on one of my servers....

Yes, my raid 5, just a test raid 5 with three really bad recycled SAS drives in it has failed; this doesn't surprise me, but it does delay me because I now have to rebuild the data... However, I know my data is good.... Lets see how good my back up is.

This back up is coming from a DD created raw image of the virtual disk, stored to and soon lifted from my NFS accessible ZFS mirrored back up server.

Therefore you would be right to ask, why are you rebuilding the virtual RAID disk in the above screen shot?  Well, I'm going to test my back up strategy!

I popped the known bad disk and the good disks out, replaced all three and I'm able to test a restore to a new virtual disk set, I have a USB boot drive ready, this is a test.

This kind of test, a real live restore, is sorely missing from so many enterprise set ups, so ask yourself is your back up going to work?