Ikea Server Rack is Finished

What kind of bloke gets to play with servers on his day off?  A damn lucky one my wife always tell me?.. Am I lucky?  Or do I have a plan?  I think I have a plan, this is a project I've been meaning to sort out for quite some time, now I have a week off of work, I'm knocking things out the park... Sooo......

Meet my new server wrack, so this is a pair of Ikea Lack Coffee tables, one with the feet fitted, and it's screwed down onto the top of another, with a pair of wheel which I also got at Ikea.

I've reinforced the leg uprights, and I'm busy fitting out the rear with a dedicated switch and power setup.

It's all on wheels so it can go anywhere in my office with relative ease.

And yes, these are my older Poweredge servers (from my fan/noise reduction videos).

Only the top unit is operational, with all the drives fitted and a full compliment of memory (64GB I think is in it).  Its socking a quad port gigabyte Ethernet at the moment but I have a whole box of fiber channel cards to go into the machines up here taking me up to 4Gbit per card (full duplex) and I plan to fit two cards in each machine so we should bond up to about 8gbits... We'll see, waiting on the cables to arrive from Hong Kong.

My other item is to get myself a dual Socket 1366 based server soon, just to play about with 6 core 12 thread CPU's (so up to 12 cores 24 threads) and perhaps in the new year I'll go up to the E5- series of Xeons to massive amounts of threads.

At the moment though, load balancing various things on my AWS instances and dropping to my own hosting is more my interests...

And a friend of mine has reminded me of a project he and I need undertake - watch this space.

Server Admin : How Good is your Backup?

How robust is your back up solution?  Go on, be honest with yourself, how good is it?... Because I've seen a whole host of them and, at this very moment, this is the screen up on one of my servers....

Yes, my raid 5, just a test raid 5 with three really bad recycled SAS drives in it has failed; this doesn't surprise me, but it does delay me because I now have to rebuild the data... However, I know my data is good.... Lets see how good my back up is.

This back up is coming from a DD created raw image of the virtual disk, stored to and soon lifted from my NFS accessible ZFS mirrored back up server.

Therefore you would be right to ask, why are you rebuilding the virtual RAID disk in the above screen shot?  Well, I'm going to test my back up strategy!

I popped the known bad disk and the good disks out, replaced all three and I'm able to test a restore to a new virtual disk set, I have a USB boot drive ready, this is a test.

This kind of test, a real live restore, is sorely missing from so many enterprise set ups, so ask yourself is your back up going to work?

Developer : How Big Is Your Floppy?

What are they teaching kids today?  I've just had the pleasure of talking to a gentleman whom has received a degree in computing from Stanford in the United States, I'm sure he paid a fortune for this education... He didn't know how many bytes were in 1.44 megabytes.

He literally said "about 1.44 million".  This was related to an old format of file being taken and converted for transmission over Wifi, the size of the data was relatively immaterial, but the source; historically speaking; was a 1.44 megabyte HD floppy disk, and though the floppy was long gone, the format of the file with it's limitation was not.

It got me thinking though, he was sat there, I'm sure earning more than myself, and he only had a jogging mental map of memory sizes, so I took a look at the Standard website, particularly the CS101 course... sure enough:

• "Byte" - unit of information storage
• A document, an image, a movie .. how many bytes?
• 1 byte is enough to hold 1 typed letter, e.g. 'b' or 'X'
• Later we'll look at storage in: RAM, hard drives, flash drives
• All measured in bytes, despite being very different hardware
• Kilobyte, KB, about 1 thousand bytes
• Megabyte, MB, about 1 million bytes
• Gigabyte, GB, about 1 billion bytes
• Terabyte, TB, about 1 trillion bytes (rare)

(Source: https://web.stanford.edu/class/cs101/bits-bytes.html)

Yeah, a terabyte is rare?... NOPE!... A kilobyte is not "about" anything it is exactly 1024 bytes, and a megabyte is not about 1 million bytes, it's exactly 1024 kilobytes, therefore exactly 1048576.  By using this "about" prefix they had made a megabyte sound 4.8576% smaller than it actually is, that's not insignificant!

And so forth.

This looks very much like an out of date page, but it is the first search result from google too!

Even wikipedia gets the right values before this page did!

WarThunder : BF109 - Stealth Damage Model Change

I would swear that there has been a stealth change to the damage model of the BF109... And a change at the fundamental level, take a look at these three screenshots....

They are three different fly outs...

Do you notice the common element?... yes, a wing tip missing, the spar exposed - but still present?.... and the aircraft can still be flown and controlled, but you are counted as shot down!... In the lower screen shot I actually held that E4 straight and level for five minutes and nearly managed to hold it til the end of the game!

But, this damage seems to come out of nowhere, and always against low-calibre fire!  It's almost like Gaijin have decided "Take a low cal hit in that box, and your wing tip falls off"...

There's no pattern to it, no reason, I've has it from behind, from head-on, from oblique angles, even from passing bombers!... BOOM no wing tip... 

And it's such a sudden change in and stark alteration of the damage model...

General Update

I've been a busy boy this weekend, I've ordered some new network wire to bring Cat 6 Ethernet directly to the upstairs bedrooms, so I have that to fit out, I've organised all my finances after the house move and everything.

I've retired my two Dell Power Edge servers - just because they were so loud - and I've got a whole bunch of pictures of them for their going on e-bay later this week - they'll be sold as a batch lot, as I basically use one as parts for the other, but they're not shabby, the heavily used on is is ironically one of the cleanest machines I have.

Their replacement is a much more suitable Pentium 4 PC - my old World of Warcraft & Eve-Online machine in fact, it's a massive downgrade in power - we're going from dual 64bit Xeon's on a server chassis to a 32bit 2001 Pentium 4.  But it's all about getting away with leaving the PC running in the front room, where it'll be part of my internet provision of services.

I've also been servicing and modding my Saitek X45 joystick, I've got a friend of a friend sorting out a set of new springs for me, I've looked carefully at modding the handle, and also had the thing in pieces to discover what's inside.  I maybe putting together a whole post on this alone, as there are not many or much good information about this solid old stick on the interwebs.  Mine being 12 years old it's lasted a long time with no intervention.  It's played me through MS Combat Flight Simulator, the original IL2, IL2 Forgotten Battles and IL2 1946.

It started to get a little funky in War Thunder last month, I noted I could not keep my 109's from pulling oddly, I thought it was a trimming issue, but it was actually the lack in my spring.

Anyway, as I say, more about all that later.

In Elite-Dangerous, I've been out and about in the universe, I've taken my pilgrimage to Sol, I arrived there last night, and docked at Abraham Lincolm station just as Australia was coming into view below.  I'm going to take a better look tonight.  However, on the way, though I have a fuel scoop I did decide to stop off and dock at a station, just to update my save at that point.  The Commodity reading software worked a treat, even on the system name it had never been trained on before.  Going forward I need to improve the capturing sequence, as it's a little clunky even for me; and I created it; but data capture is working out really well now.  Capturing and the reading of the commodities, with my myriad of little updates and string swaps is working extremely well.

Aside from changing the capturing to be more, shall we say, slick; I'm also going to be adding threaded operation to the OCR, so as I can leverage some of the multi-core features on machines to speed up the reading process.  There is a notable delay at the moment between the images being captured and the OCR kicking in.  Yes, part of this is writing the images as bitmaps to disk, but the major delay is the serial nature of it going down the list, I may as well go down two different parts of the lists at the same time!

Apart form selling the Dell servers, I'm also going to be selling a really nice 1024x768 resolution CRT monitor, my last CRT monitor.  I'm only selling it because it's being retired along with the servers, it was their heads up display.  But it's one of those crystal clear displays perfect for coding with.

I'm also going to be selling the Samsung CLP-320 laser printer, because... well, despite being only 4 years old it's not survived, it still works, but there are fading in and out on the printing, I've replaced the toners (at a hell of a cost) and so I can only conclude it needs some tinkering with or servicing, which I don't want to do.  So I'll go out the door for spares & repairs on ebay.

If I get a decent price for all these bits of stuff I'm selling, I may be looking to get myself the £160 X55 Rhino flight stick, but this is perhaps a pipe-dream with the financial situation as it is.

Elite Dangerous - Trading Tools - Client/Server - Progress Pt8

Support this project now at Patreon, or donate directly via PayPal:

The development plan, now that we have reliable data collection, is to start to share the tool; but more importantly to share the data.  I will be publishing clearer details of the XML format being used, however, it is a node only based text format, at present a commodity looks like this:

I plan to wrap this basic format into an outer "Commodities" node, with the date & time included, and make that publically available.

However, I do not plan to generate that whole list - especially as the data set grows - I plan to generate that list maybe weekly at most, as I am going to be driving this for free*.

To recap the current process for capturing the information, you play the game, and run the image processing application, then from within the game client - without tabbing to the image processor, you just press keys - you can send commands to the image processor to capture images of the navigation and commodities screen, to start them capturing the character information with tesseract:

Armed with your local data, you then run the separate windows application - and yes you do have to be able to tab to this application - to perform searches of your collected data store, which looks like this:

If you opt into sharing your data you are helping the project, at the most basic level, providing your data you will help yourself and other players!  So by sharing your local data store it gets uploaded - periodically - to the server.

Once you run the trade analyser, when you've been sharing your data, it will download the latest public list and start to use this, if you are a special supporter, sponsor or donor this is where your live cycling of the data will take place, the trade tool will automatically authenticate with the server, and start to receive the updates from all other players faster than the general list.

Of course, you will be able to write your own tool to access this same data, and be able to call the server to download the current public list of commodities and use that in your tools, other sites and projects are welcome to use this data as we collect it.  However, I do ask that you share, and help this project collect data in return.

I will be generating the special patron, sponsored, active list at least daily, if not hourly, so anyone supporting the project through Patreon, or through a sufficiently high single donation will be given access levels giving them near live, or as I say at least daily updates of the information from the project.

This won't does not affect your local data store, if you collect data live, then it is your data, and you opt into sharing that information.

As the  project matures I also hope to be capturing system connectivity data, this is where other information - such as your ship's current jump range - may be required, so help the server rationalise the data.

The server will also afford a level of checking and sanitisation to the data, so signing up to help the project you will receive a more accurate data set, not only for the commodities names, but the prices as well as the supply & demand.

What do I mean by sanitised data?  Well, the OCR is not perfect, as a side effect of either over-training, or under-training, it makes mistakes.  It also makes mistakes when the background behind your commodity screen changes - this can be because you were using the capturer whilst in the hangar, or whilst just on the surface of the landing pad.  These variations affect the isolation and interpretation being carried out.

Here is a good example:

Clearly the reader got it wrong on at least one pass, and created a new erroneous commodity, or sometimes it just plain gets it wrong:

You can spot these kinds of mistakes in the other screen shots I've already published:

The server however will be able to sanitise this data, that is correct spelling mistakes, or ask the master list (or myself) to confirm what is meant by a new name or a miss-spelled name.  Helping correct this data.

Your local data store will not do this and such mistakes will just sit there, until removed or updated by your sharing your data and receiving the new ratified list of commodities.

As well as mistakes with the name, sometimes it also makes mistakes with the values being read, this can lead to interesting problems.  I did once set off on a 4 system flight loaded with (you know I forget what) but it had a profit of 800,000 credits!!!!  WHAO!

Only when I arrived did I find out I'd made a loss, because I'd bought these things at something like 2,000 units, and the OCR had mixed up the 1877 of the destination with 7811, vasting increasing the profit margin the tools thought I would get.

This problem has been reduced, but again depending on the background at the moment the commodities or captured, of if they are a "good" screenshot or not it can happen that mistakes are made.

By sharing the data however, there will be multiple, perhaps even tens or hundreds of imprints of the same commodity at the same station coming in, the server will be able to average or at least rationalise these values and filter out wildly high values!

This is data sanitation, filtering out the outrageous mistakes we as humans would spot instantly, ornot even make as we can read the screen perfectly well.

Of course, running such servers is going to be costly, and so to share we do ask you help support the project.... Over at patreon you will find our page.

Elite Dangerous - Trading Tools - OCR - Progress Pt7

Help Sponsor me and this project now! Click here for details:

Over the weekend I've been working on the collection of data, so I've been to a few more systems and stations, I've been checking the algorithm for finding the best route, and I've been giving myself some feedback...

The first two things I immediately wanted was the program to remember the last folder used, and also as I captured data, to be able to refresh the loaded information without having to close & open the tool again.  Basic stuff.

So, loading the data looks like this now:

Lets say we've got courier missions from LHS 1914 to FK5 2550, we select the start system, so LHS 1914:

This instantly populates the Stations and total list of commodities in columns 2 and 3 there.  We can select the specific station we're docked in as well:

Reducing the number of commodities, now we could just select Fk5 2550 from the destination system list, just to see what would bring us the most profit in running into that system:

The answer here is then given in the top right box, that result will stay there now until we change system or destination system, we can get the same search just without the station select from the second column also, but then the search would start from all stations in the source system on the left.

This search alone wasn't enough however, because I needed to be able to quickly find a supply of certain commodities, hence the new controls in the right bottom.

Lets say we want to find Beryllium for a specific mission goal, we can select that commodity by name from the drop down:

The results are the source of Beryllium from the database in best price (for purchase) order.  Obviously, if you've selected a commodity you've never seen before then you get no reply, unfortunately in the current context there is no distance, or connectivity in the database, so we don't know if these items are close or not, that is still a player decision*.

The other feature added there is the "Demand For" search, this simply lists those commodities in the database which are wanted somewhere, the results are given in best price (for sale) order, this allows you to quickly marry up any profit (if any) there maybe to be had, I find it most useful if the upper algorithm driven box comes up with an illegal selection.

* However, I do envisage changing the data collection so as data is captured from system to system we have a basic web of where it connected to where by input, this basic webbing could then be used to add "how many jumps" a commodity is from the selected source or destination systems.  However, to add all this functionality I need your help!