File Catalogue

I need to write myself a piece of software to catalogue all my files and mark/reduce duplication and then store them forwards onto my back up medium.  For you see, I have a back up routine, unfortunately since 2016, I've updated this three times... And changed the disks in one of them once, this means I have four copies of my back up.

Not bad, but also not storage efficient, and I want to reduce this to two, one on site disconnected and one off site disconnected, with a buffering server in my office for fast retrieval.

To do this, I first need to know what I have.

And I have a lot of files, at least 14TB in the office alone, split between the 500GB mirror ZFS pool, the file server itself and my local drives, upon which I reckon there's at least these 3 or 4 multiple copies of everything.

There was also an old machine which had 500GB of storage for family photo's, but I've already migrated that to cloud storage.

So we're purely talking about files I have, project files, videos clips, edits and lots of code.

Now, my original plan, when putting the new PC together was to migrate a single copy to the dual 4TB drives I bought, and from there split between the new local file server and one remove cold storage server.

However, I never got around to this, I started, but didn't finish.... and in the ensuing half year I've of course used some of that 8TB of spare space for things further exacerbating the task a head of me.

NEW plan therefore, write a cataloging software tool, to parse everything, get me a checksum and hash of the file contents so I can uniquely identify each file, boil all that down into one copy on one of the 4TB drives... And over spill if I absolutely have to onto the other.

Then, spew this backup blob onto the external storage, I'm thinking of using the garage file servers I have and an Amazon AWS instance or some other cloud solution, that'll be a mammoth upload, but crucially I have time to identify the location over the next day or so whilst I write the cataloging software.

The next step will be to keep only the active projects and use the file catalogue, even expanding it with metadata, to make the remote stuff easy to parse.  One tool I'm already thinking of adding is a cpp source tree parser, to give me the namespace, class/struct names and include file pattern for any cpp file it finds, and build a DOT file graph of the header includes from a project so I can see what it is without retrieving the files themselves.

This makes it a complicated task, and it's compounded by the sheer number of files and diversity of projects and tech tinkering I've undertaken and generally left to languish as I've ran out of time or simply moved onto other things.

This whole effort is to move my code bases towards completing tasks on my own terms, my own project, my own way, and in my own time, but finishing them, because at the moment at work I'm not in that kind of control and I miss it, I miss being in control of the projects I'm working on... Hey ho, on-wards and upwards and all that.

PC Overclock and Boot Issues

I've had a bit of a rough ride the last few days, my main workstation PC has been playing me up chronic.  The first signs of problems were a failure to post, then I had a VGA missing error in quick succession.  Power cycling at the wall I got a post but the drive order has been lost.

My immediate thought was perhaps the graphics card had worked loose, I ensured it was seated and rebooted, all seemed fine.

However, getting into windows I immediately felt a sort of sluggish feeling, taking a look around I quickly saw the problem I'd lost my overclock...

To remind you the machine is sporting a Xeon X5670 overclocked to 4.0ghz, but it was running at a stock 2.53ghz, and I could really really tell.

So, back to the BIOS and I suspected I'd lost the overclock and the drive settings because of a flaky motherboard manual.

Swapping the battery out and reapplying the overclock and everything seems fine, I've even reset which drive to boot from as I'm not booting from an M.2 PCIe NVME.

But it has been a trying time, and it's really set me back... Watch this space for a 6 core, 12 thread potential replacement...

My Ultimate Upgrade

Its that time, I have been maxing out my machine repeatedly, I had two options upgrade or build new.

Money determined that an upgrade is all I can afford, between the server rebuild earlier in the year, building myself a new rack-mount table and just generally being skint my budget was low, super low, like £200.

Lets go over what the machine is....

I'm running an Intel Core i7 950 3.0ghz.  This is a four core (eight thread) chip, it cost the earth when I bought it as an early adopter of the newly released Core i7 series.  My upgrade comes in the form of an Intel Xeon X5670, clocked slightly slower I do plan to overclock the chip, but it lifts me to 6 cores (12 threads).  There's also a whopping increase in L3 cache, I see this as the last and only upgrade path for this machine.

The motherboard hosting this is a very trusty Asus P6X58-e, I've loved every minute of this board, and hope that applying the overclock will be as easy as everything else working with it.

As I am overclocking however, it's going to be a full tear down of the machine, a full clean, replacing the heatsink pads & compounds throughout and cleaning it all up.  This will be the first time I'll have taken the motherboard out of it's residential case since I built the machine.

For RAM I've been very happily running 6 x 2GB Corsair XMS3 DDR3 at 1333mhz, actually underclocking the memory as it's 1600mhz rated, but running it any faster than 1333mhz I ran into heat issues and instability.

With the rebuild however, I've bought second hand sticks of the Corsair XMS3 DDR3 but 4GB sticks, since I want to retain triple channel functionality I want six matching rated sticks, however, the number of sellers with matching sets is hard to find, there's lots of XMS3 about, but you get a speed spectrum from 1600mhz to 2000mhz stick, I've gotten some listed as different versions too, so I'm hoping everything will be okay and I can homogenise the sticks I have.

The power supply remains the Corsair I have had a while, but I am going to try and clean it out.

Brian over at TechYesCity on YouTube, often recommends using WD40 to coat parts and keep them working longer, I may have to check this on my secondary Core 2 Duo machine to see if it works before going for it on this my main machine.

The final upgrade is to sort out and replace the filters with something high micron rated, HEPA filter pads across the bottom.

WD Disk DOE & Dust....

I recently ordered a new WD Blue disk, it duly arrived and had the strangest problem I've seen in many a long year, no matter what I did the disk showed as 64MB total and it lost all data upon power cycling.

I puzzled over this briefly, until I pulled the disk back out the caddy and saw it had exactly 64MB of cache, yes, no platter space was being shown, just the cache showing up as disk space.  I have never ever seen this before, cache has always been a transparent intermediary for me in production and so to suddenly have the cache being the only exposed space was odd.

However, dead on arrival hard disk, a quick RMA and a new one winged its way into my hands, they actually gave me a slight upgrade as I got a WD Black by return of post.

Tonight I planned an hour to fit the new disk and add it to my workstation.... This machine has no cable management, and its a nightmare to add drives, there are six built in 3.5" bays but they have a set latch and screw mechanism, which forces you to fit the drives to that the power connectors are actually straining the distant power plugs. (I do plan to buy a pair of SATA power extenders on pay day).

What struck me as a massive problem however, is the amount of dust inside there.  Clearly, since fitting the 1080 GTX I've had something change or fail or open as there are copious amounts of dust in there.

Now my work area is not optimum for a workstation, I have a full tower case, so it has to go under the desk and the floor is carpeted.  Now this had been mitigated by the Coolmaster Cosmos 1000 case having lifting rails, taking the whole thing 1.25" off of the floor level, and two ground level filters, as well as a rear and front filter set.

But clearly things have not gone well, as just dust dust dust.

My original plan for the start of the year was to re-case the server, then re-case this PC as a server as well, retire one of the aging DELL 2950's for this machine itself, and put myself a new workstation together for my 40th.

Trouble is?... I'm skint, like proper skint, so I can't afford the workstation I was planning on...

And now I need to sort out this machine... I smell another re-casing task a head... Hmm, which case?  I'm thinking about the Corsair Obsidian range... Anyone have any other suggestions for a full height tower?... 1 x 5.25" at least 6 x 3.5" internal mount points and at least 2 x 2.5" mount points internally.  Front Panel USB would be a boon.

Dust seems to be my enemy.