Elite Dangerous - Trading Tools - User Privacy - Progress Pt10

Update: You can now support this project at Patreon!

The boring bits of the Elite Dangerous Trade data tool... Yes, this weekend I've been checking as to what information I can extract without wanting to draw the ire of Frontier... Using OCR you can pretty much read anything, at least here in the UK, so long as I don't represent this as my data and I don't charge anything to access the information.  Which I never will.  Then we're free to collect information.

I believe tools like this go one step further, because you can argue we'll be showing off the internal complexity of the markets in the game and so perhaps selling the trade side of the game.

Anyway, as part of the boring bits, I also have to check with you, the users, what you want to let the program do.

Now clearly, by downloading the program you accept it's going to run on your computer... You do get that yes?... I had one guy, well lets say, he's a bit slow, he didn't realise I was writing a program, he asked to be a tester and when I handed him the exe I got this very puzzling reply... "Where's the script"... it's a program mate.. a program... an executable.

Anyway, the one thing you guys get to control is what, if anything, the program is going to do with your data, both the information - like your name you type in - and the actual trade data you collect.

So, I've got some questions being asked, and a new configuration file which lets you take control of whether you share your data etc....

This is the penultimate build which is behind closed doors, I'm soon going to be able to collect this data to my server, so we'll start to see the application out there for download and more of you to try it.

However, I do still need more testers... Comment below to volunteer!

Update: You can now support this project at Patreon!

Elite Dangerous - Trading Tools - Client/Server - Progress Pt8

Support this project now at Patreon, or donate directly via PayPal:

The development plan, now that we have reliable data collection, is to start to share the tool; but more importantly to share the data.  I will be publishing clearer details of the XML format being used, however, it is a node only based text format, at present a commodity looks like this:

I plan to wrap this basic format into an outer "Commodities" node, with the date & time included, and make that publically available.

However, I do not plan to generate that whole list - especially as the data set grows - I plan to generate that list maybe weekly at most, as I am going to be driving this for free*.

To recap the current process for capturing the information, you play the game, and run the image processing application, then from within the game client - without tabbing to the image processor, you just press keys - you can send commands to the image processor to capture images of the navigation and commodities screen, to start them capturing the character information with tesseract:

Armed with your local data, you then run the separate windows application - and yes you do have to be able to tab to this application - to perform searches of your collected data store, which looks like this:

If you opt into sharing your data you are helping the project, at the most basic level, providing your data you will help yourself and other players!  So by sharing your local data store it gets uploaded - periodically - to the server.

Once you run the trade analyser, when you've been sharing your data, it will download the latest public list and start to use this, if you are a special supporter, sponsor or donor this is where your live cycling of the data will take place, the trade tool will automatically authenticate with the server, and start to receive the updates from all other players faster than the general list.

Of course, you will be able to write your own tool to access this same data, and be able to call the server to download the current public list of commodities and use that in your tools, other sites and projects are welcome to use this data as we collect it.  However, I do ask that you share, and help this project collect data in return.

I will be generating the special patron, sponsored, active list at least daily, if not hourly, so anyone supporting the project through Patreon, or through a sufficiently high single donation will be given access levels giving them near live, or as I say at least daily updates of the information from the project.

This won't does not affect your local data store, if you collect data live, then it is your data, and you opt into sharing that information.

As the  project matures I also hope to be capturing system connectivity data, this is where other information - such as your ship's current jump range - may be required, so help the server rationalise the data.

The server will also afford a level of checking and sanitisation to the data, so signing up to help the project you will receive a more accurate data set, not only for the commodities names, but the prices as well as the supply & demand.

What do I mean by sanitised data?  Well, the OCR is not perfect, as a side effect of either over-training, or under-training, it makes mistakes.  It also makes mistakes when the background behind your commodity screen changes - this can be because you were using the capturer whilst in the hangar, or whilst just on the surface of the landing pad.  These variations affect the isolation and interpretation being carried out.

Here is a good example:

Clearly the reader got it wrong on at least one pass, and created a new erroneous commodity, or sometimes it just plain gets it wrong:

You can spot these kinds of mistakes in the other screen shots I've already published:

The server however will be able to sanitise this data, that is correct spelling mistakes, or ask the master list (or myself) to confirm what is meant by a new name or a miss-spelled name.  Helping correct this data.

Your local data store will not do this and such mistakes will just sit there, until removed or updated by your sharing your data and receiving the new ratified list of commodities.

As well as mistakes with the name, sometimes it also makes mistakes with the values being read, this can lead to interesting problems.  I did once set off on a 4 system flight loaded with (you know I forget what) but it had a profit of 800,000 credits!!!!  WHAO!

Only when I arrived did I find out I'd made a loss, because I'd bought these things at something like 2,000 units, and the OCR had mixed up the 1877 of the destination with 7811, vasting increasing the profit margin the tools thought I would get.

This problem has been reduced, but again depending on the background at the moment the commodities or captured, of if they are a "good" screenshot or not it can happen that mistakes are made.

By sharing the data however, there will be multiple, perhaps even tens or hundreds of imprints of the same commodity at the same station coming in, the server will be able to average or at least rationalise these values and filter out wildly high values!

This is data sanitation, filtering out the outrageous mistakes we as humans would spot instantly, ornot even make as we can read the screen perfectly well.

Of course, running such servers is going to be costly, and so to share we do ask you help support the project.... Over at patreon you will find our page.

Elite Dangerous

I've not quite melted yet, though I'm close...

Been trying to stave off my hunger to explore the game with the older game, but as you can see I've been watching Elite Dangerous streams on Twitch.

This stream is from the venerable Andy over at "LondonGaming4Fun", I recommend his ED coverage.

Elite Dangerous - BETA 3.0 Servers Down

"We have all the major problems of a theme park and zoo all rolled into one"

Yes, its a quote, and an apt one today, as Elite Dangerous (BETA) 3.0 has hit the downloads, and even with it being a BETA Frontier Developments have two great big things in their favour when it comes to provisioning their servers for the data load they're going to take....

They know exactly how many people have bought into the game... And they can control how they turn the servers on or off.

Now, I accept they're not expecting everyone to play at once, but with a major update which they've been touting for days, if not weeks, and everyone being so intense about playing the game they've bought into then they SHOULD have expected a massive data load demand today.  Extra now and then level off when the demand plateaus, that's standard fair.  And sadly something one would hope was catered for, but which has not been.

Then the control they  have, it seems so many games design their connection mechanics around a point of access and server, so either some log-in and then direct to the data stream or a direct connection to the server, the former allows more spreading of the load, but only if the log-in server can keep up with demand, the latter opens a huge can of worms but should with scaling work more consistently more quickly.

So why does this bother me?... I've not even bought the game?... Well, it bothers me because too many people, too many players, who have paid up more than the release date cash to play are accepting this is okay, this is a beta, they cry, this is just a test, yes it is, and it's a failure, just like the test at beta 2.0 release was, there has been no difference or improvement if servers went down then and go down now, the server is the beating heart of most all the Elite Dangerous features which push it beyond my beloved Frontier Elite II... But if it is not stable, if it has no short term flex how can it flex in the long term?... Longevity, I return to it again!

I learned last night that yes you can play single player, but you still need to validate against the server, and the server was down... So I'm potentially buying an Elite game which has a finite life span!  They turn off the servers, I can't play!

I still have my complete box including 3.5" diskettes for Frontier, I can still pull an Atari ST out its box and I can still play it... If I get the Elite Dangerous : Mercenary release... I want to keep it in its box and keep it as long and I want it to be as good a game (which I don't argue it is, a far better game already) but I don't want to beholden to a server which can be shut off.

As I've previously mentioned David Braben has had his ups and downs with the Elite franchise, its ended up in and out of court, we've waited so very long for a new Elite game, and it looks brilliant, hats off to them, but for my money I want it to last, 5, 10... 15... 20... how about 35 years... Will the company still be in being?  Will the servers still run?

We've all seen what happens to Space MMO's when they turn the server off... So, longevity for Elite Dangerous... Or, Frontier Developments, let us run truely stand alone, to play single player on our own machines alone.  Yes validate stuff earned in certain single player mode before it can be moved into multi-player, I understand that, but I want to feel alone, and in awe of the sheer scale of this game, not beholden to a server with more cuckoo spit & duct tape holding it going.

Elite Dangerous - Longevity

I've said many things about Elite Dangerous on these pages, and I keep going on about it, so I have to admit I'm interested.  I'm not happy about being interested, but one just has to watch the game being played or even see screen shots and you're instantly engaged.

As an old Elite hand and a Frontier Fervent Fanatic I have a series of expectations about the game, and in comparison to Eve-Online I had to admit the idea of paying once and not being a subscriber appeals.

One of the main problems I had with a subscription is that as a married man, working and trying to live life, the amount of game time I get is quite limited.  In a week I might only get to play an hour or two, so paying to subscribe to a game for the other 166 hours that week annoys me, it is just such a waste.

I however do understand why companies need to fund their service, they need to pay the developers yes, but they also need to pay to keep the lights on, and server bandwidth is not free.  It is cheap, but it's not free, energy is very costly also.

So, they're running their game server and you're paying.

However, this isn't the funding model Frontier Developments have seemingly declared for their title, they've stated that you will pay once for the title and play there after.

The multi-player information then does state that the connectivity will be a mix of peer-to-peer and server infrastructure, this makes sense as all the dynamic market elements and what players know as "Open Play" must have some short of server back-bone, despite what some players would lead you to believe.

My question to Frontier Developments, and I have posted this off to them, has been "How in the long run do they plan on funding the server", essentially I ask because the kick-starter funding literally funds the development to a point, and the amount funded is not huge.

One pays for the game also at purchase, either now for £50 in BETA or later for £39.99 (estimated), but that's it?

I can not really understand how a server is going to be funded without some sort of revenue, players will stop joining, sales will plateaux and fall.... So, might more micro-transactions appear?... Just as there are currently skins for sale?... Might a subscription be introduced for "Open Play"... Perhaps the game falls down to Single Player only, or Group play only?...

Personally I miss my adventures hosting the Frontier Club, where peeps sent in their disks with saved games, I copied them back to everyone else's disks and mailed them back... Good fun.

Elite Dangerous - Still Not Going To Play It

I am not going to play Elite Dangerous... Let me get that said, not because I have to eat humble pie and say it does not look a bad game, and unlike my prior posts David Braben has proven himself to NOT be a snake oil sales man, as he has delivered the game to the masses (least as a test/alpha/beta so far)...

But, that said, watching someone competent play is interesting, prior to seeing the below videos I've only watched some mediocre coverage from Jingles and then the terrible developer example videos from back when they were still looking for online funding...

Ralfidude however, a YouTuber with a plethora of DCS and other flightsim coverage, has taken to the space ways, and he's given us these few videos with his coverage of the game... Looks good.

My problem with the game is not the game, I love Elite, and I understand it there is a single player game involved in this title... However, its the online play which counts and it seems to me there are too many ways for people to grief.... And I did all the griefing I needed in Eve-Online.

I'll admit though, Ralfi's Videos have me tempted... But not for £50... Fuck that price point with their millions upon millions start up capitol already in the bag!

World of Tanks - Grinding to the T34

I've reinstalled the machine with Windows 7, and sure enough all the networking problems I was having have gone away, but I'm pleasantly surprised with how spritley the machine now is.

Vista must have been really really in efficient, for example, running World of Tanks, I would run her only in medium to low settings to get 20 fps...

With no changes of hardware, just an update to Windows 7, and I'm now in 1680x1440 resolution, high settings for everything and I get 50-55fps.

Its so pretty....

That was the main victory of the evening, which I spent enjoying with lots of rounds of World of Tanks.

My first discovery was something I had spotted before, but I think will take the next patch to explain, you see you now get missions in the game.  Or at least the rewards for the mission, but you don't have any way of knowing what the missions you have are.  They maybe listed somewhere, but no-where I've spotted in game.

In patch 8.8 however, which is coming soon, WarGaming are adding a way to spot what the missions are.

The night however, as enjoyable as it was, did result in a spectacular number of Defeats for me, despite my best efforts.

Sometimes I was simply out played, sometimes I was clearly stuck into odd matches, and I was spending a lot of games in my T-80 Light tank - an odd machine, grinding the XP for my T34.

But my efforts were not helped with derps like this

Or grease balls like this, whom I spotted on numerous occasions sat in strange locations usually at the back of the pack waiting to pick off enemy targets which the rest of the team - whom are lower tier - have softened up for them... Basically reaper creepers, that's the term I'm going to invent for them.

I did get my T34 before ceasing playing for the evening, and I even unlocked the high-pen gun, but I'm not happy with the vehicle, its match making is pretty terrible.

And I'm also stuck with the news that my favourite tank - with my most trained crew - the VK30.01H is about to be turned from a tier 6 medium into a tier 5 heavy?  Go figure, Jingles has made comment about this, but... Argh, I just want to smack WarGaming.

It might be time for a break and return to war thunder, in which I have also been playing recently, and made a level with my Japanese aircraft.

Also still enjoying Elite Frontier...

Long and Fraught

It has been a long and frightfully fraught day, tomorrow can only get better... We're off out to sort out a new mortgage for a house we're buying... which will be fun.

But, for entertainment this evening, I've left off with my World of Tanks kicker, and I've played a couple of rounds of war thunder... I'm not sold on the whole plane upgrades thing (yet) but I'm so happy to see the oft remarked about broken economy get a fix.

And as ever even on my terrible 260 GTX graphics card on this machine that game looks gorgeous.

However, its not as gorgeous as the other game I've been playing this week...

No folks, don't adjust your sets, that is the graphics, this is actually the best graphics you'll ever likely see in this particular game, for this is Elite II Frontier - the best game ever - but this is amazingly the Atari ST version, decompiled and then stripped of ST hardware called and then converted to using OpenGL on the PC to render... an amazing piece of programming which I can barely get my head around - despite having downloaded the code and had a look - and also an amazing way to re-enjoy this most classic of games.

Not a lot of people know this, but I was so taken with this game that I tried to start a club around it, long before the internet, and before BBS's were buzzing about this game I send a missive off to the late great ST Format and said I'm here talk to me as the Frontier Club.  My parents went mental about this when my little brother showed them in the magazine.

They thought I would need a license or pay tax on whatever I had, so they made me turn it into a free users group, which promptly flopped.

But for a short time I made my first ever bit of cash from these computers, and that little bit of cash I made was turned into a few news letters, envelopes and stamps to mail people their disks back and forth, I had five regular commentators, and for their £5 each "member" of the original list got a couple of news letters, planned to be quarterly, they got some fan fiction, and the exchange of saved games from other players in the club.

My parents killed it tho, my little brother also, to this day - despite my working as a software engineer - they don't seem to realise how much they held me back when I was at the forefront of something popular and amazing, and making money...