Monday, 12 November 2018

RIP : Stan Lee

Stan's done and gone... Sad...


Tuesday, 6 November 2018

Hackers....

God I love this film...


Sunday, 4 November 2018

Server Crash... Bad Memory

I've spent about three hours checking and sorting out the machine which went down, and come-what may I found problem after problem, but not with any of the software.  If your software checks out then there's an obvious other place to look... Your hardware.

My first step was to remove the drives, specifically my ZFS devices.  No difference, everything still broke.

I then set about systematically removing one piece of hardware at a time, first my quad ethernet card, no difference, then I removed the added dual SATA card still broke...

Finally, I removed all the memory and inserted just one stick...

For the first time the firmware of the motherboard reported something had changed (at boot) it knew the memory has reduced from 8GB to 2GB.  But then the grub boot was really fast, it came up and into the system very quickly.

Now, I had been getting MySQL Error 2013 for nearly ever query or SQL dump, with 2GB of RAM this didn't happen, sudo and all the programs now work again... I can therefore only assume one of the other sticks of RAM is corrupt in some manner.


So with a clean boot...


I was able to start pulling the data out...


I ran the server on soak in memtest for an hour and still no problem, there was ONLY a problem after MySQLd had started and 8GB of RAM was installed... Time to bin some of these sticks.

Server Crash... And Corrupt

Yeah, something very bad has happened with one of my boxes... It was not responsive over the network this morning, I thought little of it, power cycled it and came back to it this afternoon.

Anyway, it was still not playing ball, I rolled the vmlinuz image back to 4.4 from 4.15 (the last unattended update I did last week) but everything.  I mean everything, every command, every program (except cat and ls) resulted in a "segmentation fault".

Booting into recovery I got a little further, but the syslog looks very worrisome...


And I could only see this image when I had hauled the machine back into the office and plugged a VGA cable into it, over the network and even over the serial TTY it just sat there spinning it's wheels.

I would like some of the data back off this box, it contains the current (more recent) live image of my WOW server and it contains the old archived version of my Ark server.

Other than that there is a ZFS area which I've dumped home photo's onto, but getting that back is easy, I just pull one of the drives out and remount the ZFS elsewhere... I think though I may buy a new main disk for this machine, I like it.