What’s data without some graphs? Here is a small snip of the graphing that is currently in the spotmarket project.
2016-01-13 to 2016-06-07
2016-01-13 to 2016-06-07
Jumps in Trade Hubs
Jumps and NPC Activity in The Northern Regions
Gurista Ratting Rates
NPC Universe Overview
Whew, this was a big step in the right direction. I added a lot of sanity (keep everything in UTC), usability, and cleanup to the existing design. We’ve got moongoo, the beginnings of indexes, and CREST killmail data now flowing in to the growing dataset. The backend was improved by migrating to PostgreSQL 9.5.1 and I also cleaned up the install directions to try to make it easier.
[bug] Change timestamp for zkillboard and markethistory consumer to use UTC for mental sanity
[enhancement] Add 404 error page
[enhancement] Change to postgres 9.5.1 to support jsonb
[enhancement] Parameterize graph functions
[enhancement] Change date format in graphs to ISO for mental sanity
[enhancement] Standardize table formatting
[enhancement] Supervisor to make Flask web service persistent
[enhancement] Add jsonb index on data.killmails for typeID and solarSystemID
[enhancement] Add paging to zKillboard consumer
[enhancement] Add check to resume from last recorded page for zKillboard consumer
[enhancement] Add basic exception handling to zKillboard consumer
[new] Creation of ship index report
CREST Verified Moon Minerals
It is no secret that I have an affinity for finding accurate moon data. In a previous post, I’ve gone so far as to chart the regional density of each moon mineral, showing that there were built in skews as to where each type was placed for each class (R8, R16, etc).
In the Rubicon expansion, CCP introduced siphon units, which can be anchored near a POS to slowly leech items from within the shields. When an extractor is destroyed the killmail will show what moon mineral was in it. This, combined with the x,y,z location data that started to be exposed after the Parallax expansion, can be used to locate the nearest celestial object to give us a verified report of what moon mineral is contained in the object.
I gathered some examples for anyalisis, and over a long flight, I started to put some code together to use killmail x,y,z coordinates to parse CREST killmails. After implementation and testing, it has proven to be accurate so I’ve included the feature in this release. There is a manual step to get the moon mineral data into a table where it is used in the web frontend; I wanted to keep them separate to isolate the two datasets.
PostgreSQL Upgrade to 9.5.1
Rather than writing a parser for CREST killmails, I decided to store the JSON itself for simplicity. Developing on 9.3 with the json datatype proved to be a path that I did not want to go down. 9.3 is not the prefered version and lacked support for a fancy datatype inherit to 9.4+ and above. The following blog post also convinced me to migrate.
The proof is in the numbers so here’s a side-by-side comparison of the same query on the same dataset with only the datatype being different. The two virtual machines also had access to the same amount of CPU and RAM to control the results.
This query was proof enough that I made the right choice. Later that night while checking my logging table, I also noticed an improvement in insert speed.
9.3 (json) [kills] insert 2982 @ 93.81 rec/sec 9.5.1 (jsonb) [kills] insert 2850 @ 124.44 rec/sec
Not bad for about 45 minutes of testing and correcting install instructions.
What’s up for the next release? Check out the TODO.md for a full list of items slated for each release. The main focus is going to be putting more control in the web front end, letting you enable/disable import items, add item to the market/zKillboard watch list, etc.
I published the initial code to GitHub with install directions for Ubuntu. The project so far is a Python script that consumes map API jump and kill activity and inserts them into a PostgreSQL database. Details as to why I picked these two items as starting points to come later.
So far this has been a fun learning project; I’m getting more familiar with Python datatypes/iterators/quirks, have been leveraging arrow for any datetime work, am getting more comfortable with pandas, and have a handle on the basic differences between MySQL and PostgreSQL.
Growing Laundry List
- Implement more efficient UPSERT to improve database insertion efficiency.
- Figure out what CREST market data to record.
- Start working on putting the pieces together for the first analysis — more on that later.
Presenting data with Python Pandas.
Manually extracting the data to a csv and charting it in D3.js for now.
Burn Jita got off to a great start with freighter kills coming in one after another. I was able to watch a few hours of NickFuzzeh’s stream yesterday and was able to get some nice screencaputres before it went offline; image credits go to him.
If you are looking to keep up to date on the kills, head on over to use isjitaburning.com.
CCP has released two out of the six Devblog posts aimed at industrialists detailing changes for the upcoming Summer expansion. We’re seeing sweeping changes to the way logistics are done for capital ships, station research, POS anchoring limitations, BPO security concerns, and how inventors are going to be given a boost with BPC copy rates. Lockefox has a great summary of the concerns in his Everything Is Changing post.
Given the ideology coming out of the Dev Blogs to empower Nullsec industrialists while kicking Lowsec in the knees, I wanted to see how much of my industrial gameplay occurs in Lowsec — Is CCP killing my game?
Summary of profit grouped by solar system with profit numbers obfuscated.
— Get Profit per Station By Solar System including Region
SELECT SUM(profit), mapDenormalize.itemName, mapSolarSystems.solarSystemName, mapRegions.regionName, AVG(mapDenormalize.security)
ON (wallet.stationID = mapDenormalize.itemID)
ON (mapDenormalize.regionID = mapRegions.regionID)
ON (mapDenormalize.solarSystemID = mapSolarSystems.solarSystemID)
GROUP BY wallet.stationID
ORDER BY sum(profit) DESC
It turns out that Lowsec only accounts for 7% of our profits to date so I can’t complain about the nerf that is going to hit Lowsec capital builders given the upcoming compression changes.
The changes to compression are a welcomed change, even if it means retiring or heavily modifying the logistical chain for Lowsec capital production. I have a feeling that there are going to be more major changes in the next four upcoming Devblog posts. Stay tuned.