Evemail from Gerard Nardeaux

I have not been keeping up with reading evemails to Blake Armitage as he has been grinding longer skills. As a result I have not responded to some really thoughtful evemail inquiries.

I now want to take the time to respond to a few of them. I contacted Gerard Nardeaux and he gave me the green light to post my responses on the blog.

2013.06.03 17:36
I am desperate to understand how you organize and use market and industrial data. EMDR is a great tool, but I personally have no idea how to parse through the literal fire hose of data and then load it into MySQL for clean price data. As a result, I’ve been taking advantage of sites that regularly update EMDR data. Do you use MySQL to sort through historical data to produce weekly, monthly, quarterly, and annual reports? Or do you manually keep track of expenses and revenue? As of right now with a total of 8 research and manufacturing slots, this is no problem for me. I imagine it gets very difficult as the scale of production increases, though.

All of my graphs are produced in Excel with CSV exports from phpMyAdmin that is connected to our database. I aggregate records of every transaction in a simple ‘wallet’ table that is fed from a few character’s API’s. I then perform some simple data grouping or sorting in my initial query and then take that into Excel to make a user-facing graph or table as it makes it look more polished.

Outside of the normal SELECT and FROM statements you can leverage a lot of power by using simple WHERE, ORDER BY, or GROUP BY statements:

• To take a look at items in 2012, use WHERE transactionDateTime LIKE  ‘2012-%’
• Filtering only Carrier and Dreadnought hulls can be done by using WHERE typeID IN (23757, 23915, 23911, 24483, 19724, 19720, 19726, 19722)
• GROUP BY typeID can be used to gather intelligence on how items compare to eachother
• GROUP BY wallet.typeID combined with ORDER BY totalProfit DESC will let you know in Descending order which items are the most profitable

Outside of mucking around in the database tables, my partner and I use web based production and tracking sites to aid our projects.

Raath has created industry.darkshadowindustries.com, which strongly caters to our manufacturing wing, and I released my Wallet Manager that has a lot of reporting and profit tracking.

Keeping track of profits is done on a first in, first out basis as outlined here in a previous post.

You’re right to assume that my margin of error for asset tracking increases along with scale. I believe that I am currently giving myself a +/- 10B on a net worth of 200B. Right now there are assets in construction, transit, and in the wallet of a industry partner as we are constructing some large-scale items.

I recently had a discussion with Raath, who is warehousing the EMDR information, so this may help answer your question about how we handle it. We’re not using the data for historical trending, only what is the ‘now’ in New Eden.

(9:09:31 AM) Raath: the database should only have data going back 2 or 3 days
(9:09:45 AM) Raath: the point of it is that it aggregates current market data
(9:10:23 AM) Raath: but I was caching stuff from when I started catching the stream so I’ve got backed up about 6 months of sales
(9:10:45 AM) Raath: but now I hourly clear out expired orders so the only thing in our DB right now is confirmed live orders
(9:10:58 AM) Raath: and a couple of thousand confirmed expired