Elite Dangerous - Trading Tools - Office & Server Update - Progress Pt11

Update: You can now support this project at Patreon!

Where have I been?  And what have I been doing?

Well, I've not been coding, not one line of new code has passed my finger tips.

What has been happening though is a complete remodelling of my office and work area, I've been to ikea and on my zero dollar budget (yes, no-one has yet dipped into their pocket to sponsor this project) I got a very cheap new table, a monitor rising shelf and made my power arrangements safe.

This new work area has then seen over 18 hours of solid testing, from system to system, station to station, collecting data.  As well as using the analysis tool to make advantage of it.

My next quest/test will be to allow manual editing, or at least suggested editing, of the database to remove, or correct those sometimes mis-detected names of commodities.

The test after that will be to go and find some rares, see how they process out, and begin a list of them spotted, so as they can be highlighted in the analysis tool.

The data collection server is also a priority now, the old Power Edge 2650 machines have been decommissioned, and I've actually donated them to a charity so they can perhaps sell them and make some money.  The new server is a Pentium 4 based box, which will be running Ubuntu Server, or maybe Arch... Though in the long run a Raspberry PI 2 running Arch may take over, if suitable network storage can be arranged, or if I can power a 2.5" 5V HDD from the Pi itself via USB for storage.

Cheap cheap, this is the way.

I've had more correspondance from potential users and people just asking for the program, it's not ready folks, help me to help you by supporting the project with a donation, or through patreon.

As for those still asking "why", simply because I don't want to run on my computer things others have written, I find some tools collecting market data must be simply opening the Elite Dangerous process, getting the tokens set to read the memory and slurping out all the data they want.  Frontier asked other projects to stop doing that, so we won't even state.

And then why am I doing this alone?  Because I like to code, it's my profession, not just a hobby, this isn't just a script thrown together in Python, or a slap dapse executable thrown out there with who only knows what malware attached, no this is a formal project, with revisions, a coding standard and it's letting me get to grips with OCR, something I've always thought about needing, but never got to use.

More updates at the weekend folks.

Update: You can now support this project at Patreon!

General Update

I've been a busy boy this weekend, I've ordered some new network wire to bring Cat 6 Ethernet directly to the upstairs bedrooms, so I have that to fit out, I've organised all my finances after the house move and everything.

I've retired my two Dell Power Edge servers - just because they were so loud - and I've got a whole bunch of pictures of them for their going on e-bay later this week - they'll be sold as a batch lot, as I basically use one as parts for the other, but they're not shabby, the heavily used on is is ironically one of the cleanest machines I have.

Their replacement is a much more suitable Pentium 4 PC - my old World of Warcraft & Eve-Online machine in fact, it's a massive downgrade in power - we're going from dual 64bit Xeon's on a server chassis to a 32bit 2001 Pentium 4.  But it's all about getting away with leaving the PC running in the front room, where it'll be part of my internet provision of services.

I've also been servicing and modding my Saitek X45 joystick, I've got a friend of a friend sorting out a set of new springs for me, I've looked carefully at modding the handle, and also had the thing in pieces to discover what's inside.  I maybe putting together a whole post on this alone, as there are not many or much good information about this solid old stick on the interwebs.  Mine being 12 years old it's lasted a long time with no intervention.  It's played me through MS Combat Flight Simulator, the original IL2, IL2 Forgotten Battles and IL2 1946.

It started to get a little funky in War Thunder last month, I noted I could not keep my 109's from pulling oddly, I thought it was a trimming issue, but it was actually the lack in my spring.

Anyway, as I say, more about all that later.

In Elite-Dangerous, I've been out and about in the universe, I've taken my pilgrimage to Sol, I arrived there last night, and docked at Abraham Lincolm station just as Australia was coming into view below.  I'm going to take a better look tonight.  However, on the way, though I have a fuel scoop I did decide to stop off and dock at a station, just to update my save at that point.  The Commodity reading software worked a treat, even on the system name it had never been trained on before.  Going forward I need to improve the capturing sequence, as it's a little clunky even for me; and I created it; but data capture is working out really well now.  Capturing and the reading of the commodities, with my myriad of little updates and string swaps is working extremely well.

Aside from changing the capturing to be more, shall we say, slick; I'm also going to be adding threaded operation to the OCR, so as I can leverage some of the multi-core features on machines to speed up the reading process.  There is a notable delay at the moment between the images being captured and the OCR kicking in.  Yes, part of this is writing the images as bitmaps to disk, but the major delay is the serial nature of it going down the list, I may as well go down two different parts of the lists at the same time!

Apart form selling the Dell servers, I'm also going to be selling a really nice 1024x768 resolution CRT monitor, my last CRT monitor.  I'm only selling it because it's being retired along with the servers, it was their heads up display.  But it's one of those crystal clear displays perfect for coding with.

I'm also going to be selling the Samsung CLP-320 laser printer, because... well, despite being only 4 years old it's not survived, it still works, but there are fading in and out on the printing, I've replaced the toners (at a hell of a cost) and so I can only conclude it needs some tinkering with or servicing, which I don't want to do.  So I'll go out the door for spares & repairs on ebay.

If I get a decent price for all these bits of stuff I'm selling, I may be looking to get myself the £160 X55 Rhino flight stick, but this is perhaps a pipe-dream with the financial situation as it is.

Elite Dangerous - Trading Tools - OCR - Progress Pt7

Help Sponsor me and this project now! Click here for details:

Over the weekend I've been working on the collection of data, so I've been to a few more systems and stations, I've been checking the algorithm for finding the best route, and I've been giving myself some feedback...

The first two things I immediately wanted was the program to remember the last folder used, and also as I captured data, to be able to refresh the loaded information without having to close & open the tool again.  Basic stuff.

So, loading the data looks like this now:

Lets say we've got courier missions from LHS 1914 to FK5 2550, we select the start system, so LHS 1914:

This instantly populates the Stations and total list of commodities in columns 2 and 3 there.  We can select the specific station we're docked in as well:

Reducing the number of commodities, now we could just select Fk5 2550 from the destination system list, just to see what would bring us the most profit in running into that system:

The answer here is then given in the top right box, that result will stay there now until we change system or destination system, we can get the same search just without the station select from the second column also, but then the search would start from all stations in the source system on the left.

This search alone wasn't enough however, because I needed to be able to quickly find a supply of certain commodities, hence the new controls in the right bottom.

Lets say we want to find Beryllium for a specific mission goal, we can select that commodity by name from the drop down:

The results are the source of Beryllium from the database in best price (for purchase) order.  Obviously, if you've selected a commodity you've never seen before then you get no reply, unfortunately in the current context there is no distance, or connectivity in the database, so we don't know if these items are close or not, that is still a player decision*.

The other feature added there is the "Demand For" search, this simply lists those commodities in the database which are wanted somewhere, the results are given in best price (for sale) order, this allows you to quickly marry up any profit (if any) there maybe to be had, I find it most useful if the upper algorithm driven box comes up with an illegal selection.

* However, I do envisage changing the data collection so as data is captured from system to system we have a basic web of where it connected to where by input, this basic webbing could then be used to add "how many jumps" a commodity is from the selected source or destination systems.  However, to add all this functionality I need your help!

Elite Dangerous - Trading Tools - OCR - Progress Pt6

Update: You can now support this project at Patreon!

Armed with some of my data I've had two stabs at making use of the data, and so here's the first peek at the tool I've thrown together to let me make use of the information, one simply selects the start system and then the destination system and the best commodity (for profit) is calculated.

You can also refine the start point by station or specifically a commodity, before you select the destination to give a specific start point and be given the best profit item.

The next feature I'm going to add is a box to enter the name of a commodity and find where it's sold as a list of system & station names, just as a helper for what's already in the game's galaxy map.

Elite Dangerous - Trading Tools - OCR - Progress Pt5

Update: You can now support this project at Patreon!

So, last night I had my first long run and tuning up session, capturing system name, station name and commodities as I played the game... As you can see, I have quite a few captured commodities now:

I'm now going to throw together a separate reader, to load the data set and let me do comparisons or round-trip plotting of best prices.

But, I thought i'd explain how this application works, as I've posted it into the streams of a few Twitch chaps playing Elite, and everyone seems to have their own preconceptions....

1) This is a stand alone application, it runs in windows completely separately to Elite.

2) It captures the screen images, it DOES NOT open the processes memory.

3) The areas of the screens captured are filtered, and then passed to an optical character recognition (OCR) system.

4) The OCR system I'm using is Tesseract

5) The image processing is triggered by keys:  In game you just play, and then Press F1 whilst in the navigation menu, and the program will take a shot and try to work out the system.

6) Press F2 whilst in the commodities screen, and the station name is captured.

7) Press F3 and this toggles on the capturing of the area the commodities are listed in, it takes pictures until you press F3 again, so you slowly scroll down the list, then press F3 and the program passes the images taken for processing.

8) The data is written out as XML and contains the System/Station name, then the Commodity Name, the buy and sell prices, and then the supply & demand levels.

I have had to do some manual fixing of things, such as correcting for the OCR getting the strange minerals spelt wrong.

My plan once I have the data coming in, is to have it upload to a server, which I can then view from anywhere and use to make searches of the commodities.

I would however like some help completing the application, specifically I require testers, if you are interested, please contact me in the comments below.

Finally, lots of people ask "Why", why make this tool, why not use one of the others out there... Well, I have nothing against the others, I looked at a couple, including EliteOCR and they showed that Tesseract could get the job done, but none of those tools were mine, I've played and used a lot of tools for game, I'm a professional programmer, so I figured I should make a tool which is exactly what I want.

Will I never make this publically available?... Perhaps, but I've had little to no interest thus far.

Elite Dangerous - Trading Tools - OCR - Progress Pt3

Update: You can now support this project at Patreon!

Having played with my own OCR, I've decided to use Tesseract, however, I'm doing it in an unusual manner.  Because this is all really a learning exercide for me, and to keep the build small, I've put the tesseract API calls into their own EXE, not a DLL, an EXE I can call giving it an image filename and an output text path.

The output is XML, tells me the result and likely text result, handling errors with the image or training data for me and not crashing or causing issues with my application.

This clear division of effort lets me just use the tesseract api example (lepttest) code to do my OCR, whilst I get the skimming of the Commodities actually working.

Then I can revisit the OCR later, or maybe play with my own....

Here's a screen shot of the application "reading" from the screen...

As you can see, this is taking a screen capture of the area of the commodities window live from the client, and it's just updating any commodities it see's in the list...

My capturer also works at any resolution, so you can be fullscreen, windowed, or any resolution in the game, there's no need to be a specific resolution as some other tools require of the Elite client.

I am however struggling with the OCR, I think I need to retrain, or better train, Tesseract for the font used in Elite, but that's a separate problem than my actual tool program, as I've split the OCR out!

Programming - Elite Dangerous Tools - OCR Pt 2

Update: You can now support this project at Patreon!

Some progress with the OCR I'm implementing, I've now broken the black and white filtered image down into rows, and then each row into the characters, and then I've looked and analysed where the gaps are between characters and decided where the columns are....

So here we see my pattern matching (simple pattern matching - not a neural network) reading through the characters and building the words for column 1, which is the name of the commodity...

As you can see it is working quite well... I need to keep training the pattern matching weights more, and this is very memory intensive (it takes over 300MB of data to process the various images, sub images, patterns and pixel values I'm summing & weighing against one another).

The biggest stumbling blocks is that the sourcing of the images from the client is very user dependent and twitchy, there are rules for it and ways for the user to capture the image in order and with the best overlap to help the stitching algorithms.

But this is the first pass of this tool, this is for me, to help capture and use market data personally... When I've collected a bunch of such data I may see if there are interested parties in helping collect more and more data.

Until then however, keep yourselves informed by popping back to the blog from time to time, and as ever let me know if you would use a tool like this!  Or even sponsor me to develop it further!