Elite Dangerous - Trading Tools - Office & Server Update - Progress Pt11

Update: You can now support this project at Patreon!

Where have I been?  And what have I been doing?

Well, I've not been coding, not one line of new code has passed my finger tips.

What has been happening though is a complete remodelling of my office and work area, I've been to ikea and on my zero dollar budget (yes, no-one has yet dipped into their pocket to sponsor this project) I got a very cheap new table, a monitor rising shelf and made my power arrangements safe.

This new work area has then seen over 18 hours of solid testing, from system to system, station to station, collecting data.  As well as using the analysis tool to make advantage of it.

My next quest/test will be to allow manual editing, or at least suggested editing, of the database to remove, or correct those sometimes mis-detected names of commodities.

The test after that will be to go and find some rares, see how they process out, and begin a list of them spotted, so as they can be highlighted in the analysis tool.

The data collection server is also a priority now, the old Power Edge 2650 machines have been decommissioned, and I've actually donated them to a charity so they can perhaps sell them and make some money.  The new server is a Pentium 4 based box, which will be running Ubuntu Server, or maybe Arch... Though in the long run a Raspberry PI 2 running Arch may take over, if suitable network storage can be arranged, or if I can power a 2.5" 5V HDD from the Pi itself via USB for storage.

Cheap cheap, this is the way.

I've had more correspondance from potential users and people just asking for the program, it's not ready folks, help me to help you by supporting the project with a donation, or through patreon.

As for those still asking "why", simply because I don't want to run on my computer things others have written, I find some tools collecting market data must be simply opening the Elite Dangerous process, getting the tokens set to read the memory and slurping out all the data they want.  Frontier asked other projects to stop doing that, so we won't even state.

And then why am I doing this alone?  Because I like to code, it's my profession, not just a hobby, this isn't just a script thrown together in Python, or a slap dapse executable thrown out there with who only knows what malware attached, no this is a formal project, with revisions, a coding standard and it's letting me get to grips with OCR, something I've always thought about needing, but never got to use.

More updates at the weekend folks.

Update: You can now support this project at Patreon!

Elite Dangerous - Trading Tools - Client/Server - Progress Pt8

Support this project now at Patreon, or donate directly via PayPal:

The development plan, now that we have reliable data collection, is to start to share the tool; but more importantly to share the data.  I will be publishing clearer details of the XML format being used, however, it is a node only based text format, at present a commodity looks like this:

I plan to wrap this basic format into an outer "Commodities" node, with the date & time included, and make that publically available.

However, I do not plan to generate that whole list - especially as the data set grows - I plan to generate that list maybe weekly at most, as I am going to be driving this for free*.

To recap the current process for capturing the information, you play the game, and run the image processing application, then from within the game client - without tabbing to the image processor, you just press keys - you can send commands to the image processor to capture images of the navigation and commodities screen, to start them capturing the character information with tesseract:

Armed with your local data, you then run the separate windows application - and yes you do have to be able to tab to this application - to perform searches of your collected data store, which looks like this:

If you opt into sharing your data you are helping the project, at the most basic level, providing your data you will help yourself and other players!  So by sharing your local data store it gets uploaded - periodically - to the server.

Once you run the trade analyser, when you've been sharing your data, it will download the latest public list and start to use this, if you are a special supporter, sponsor or donor this is where your live cycling of the data will take place, the trade tool will automatically authenticate with the server, and start to receive the updates from all other players faster than the general list.

Of course, you will be able to write your own tool to access this same data, and be able to call the server to download the current public list of commodities and use that in your tools, other sites and projects are welcome to use this data as we collect it.  However, I do ask that you share, and help this project collect data in return.

I will be generating the special patron, sponsored, active list at least daily, if not hourly, so anyone supporting the project through Patreon, or through a sufficiently high single donation will be given access levels giving them near live, or as I say at least daily updates of the information from the project.

This won't does not affect your local data store, if you collect data live, then it is your data, and you opt into sharing that information.

As the  project matures I also hope to be capturing system connectivity data, this is where other information - such as your ship's current jump range - may be required, so help the server rationalise the data.

The server will also afford a level of checking and sanitisation to the data, so signing up to help the project you will receive a more accurate data set, not only for the commodities names, but the prices as well as the supply & demand.

What do I mean by sanitised data?  Well, the OCR is not perfect, as a side effect of either over-training, or under-training, it makes mistakes.  It also makes mistakes when the background behind your commodity screen changes - this can be because you were using the capturer whilst in the hangar, or whilst just on the surface of the landing pad.  These variations affect the isolation and interpretation being carried out.

Here is a good example:

Clearly the reader got it wrong on at least one pass, and created a new erroneous commodity, or sometimes it just plain gets it wrong:

You can spot these kinds of mistakes in the other screen shots I've already published:

The server however will be able to sanitise this data, that is correct spelling mistakes, or ask the master list (or myself) to confirm what is meant by a new name or a miss-spelled name.  Helping correct this data.

Your local data store will not do this and such mistakes will just sit there, until removed or updated by your sharing your data and receiving the new ratified list of commodities.

As well as mistakes with the name, sometimes it also makes mistakes with the values being read, this can lead to interesting problems.  I did once set off on a 4 system flight loaded with (you know I forget what) but it had a profit of 800,000 credits!!!!  WHAO!

Only when I arrived did I find out I'd made a loss, because I'd bought these things at something like 2,000 units, and the OCR had mixed up the 1877 of the destination with 7811, vasting increasing the profit margin the tools thought I would get.

This problem has been reduced, but again depending on the background at the moment the commodities or captured, of if they are a "good" screenshot or not it can happen that mistakes are made.

By sharing the data however, there will be multiple, perhaps even tens or hundreds of imprints of the same commodity at the same station coming in, the server will be able to average or at least rationalise these values and filter out wildly high values!

This is data sanitation, filtering out the outrageous mistakes we as humans would spot instantly, ornot even make as we can read the screen perfectly well.

Of course, running such servers is going to be costly, and so to share we do ask you help support the project.... Over at patreon you will find our page.

Elite Dangerous - Trading Tools - OCR - Progress Pt4

Update: You can now support this project at Patreon!

Just an quick update, since this morning, I've been tidying the code, fixing things up to make it work with the actual client, and here's the first basic run....

This is a looping console application, so we press F1 when looking at the navigation console to grab and read the system name, and when docked with the commodities window open we press F2 to grab the station name.

Pressing F11 gives the status of these two strings, and you can press Y/N to confirm the selected names before going any further.

This is the important header information, captured live from the client, the client again can be any resolution, windowed or not, you don't need to do anything special for this, just look at the right pages in the client before pressing the F1/2 button respectively, you keep the focus in your client (so you can be full screen) and my OCR application, in the background, works seamlessly.

Now, the OCR is not quick, nor is it perfect, but here we can see it working, and for my needs this is just the ticket.

Next, the actual commodities, which I'm going to save as XML files all over... And then I need to throw up a reader to take those XML files in and make a database of searchable ccommodities  The main search being you select a commodity you see on screen in large numbers, and the tool will get all the buying stations up in order of the buying price, optimizing that trade for you.

The second search I'd like is to give it say 4 systems, which I see in the client are linked & in jump range even when full, and the tool work out the best trades (top 3 maybe in case there are few of the items) to take between them in a sequence.