spotmarket – 0.3-dev

Rapid Affirmative Coding

The last four weeks have been a rather busy development cycle for me. I finished up learning the basics of Python, have a working backend database, and have become more conformable with the frontend side of my platform. I’ve restructured a lot of items and have a vision down for the final architecture.

2016-02-24_spotmarket_diagram

The end goal is to have everything in a Docker container so that anyone, anywhere can deploy; I want to aim for a ‘push-button-receive-data’ style of deployment. Let’s keep the IT wrangling to a minimum.

Currently the frontend that I have is build on Flask+Jinja2. As I have been developing the proof of concept out, I’ve found some limitations. James, who wrote the original trading program with me, has been pushing me towards AngularJS as the presentation framework.

Luckily I’ve kept to the original design principal of keeping every process that needs to communicate as an isolated component. With this design, any component can be replaced or moved and it is transparent to any other layer or neighboring service. Service ‘A’ could can be replaced with no impact to the front or backend, or the backend could be replaced with no impact to the services, and so on.

2015-12-15 stack design.png

Laundry List of Improvements and Highlights

1. We’ve Got Data

2016-03-01_db_tables_1.png

2. Improved Database Inserts

I rewrote the insert statements to use primary key constraints to perform error checking rather than selecting if the row exists and then inserting. PostgreSQL does have an UPSERT statement that I would like to utilize to gain even further insert speed, but that will have to come when I update my PostgreSQL version to 9.5+.

Before: [kills] insert 3128 @ 49.85 rec/sec
After: [kills] insert 2948 @ 94.84 rec/sec

3. RESTful API

Any presentation item that needs data receives it over an JSON API.

2016-03-01_api_1.png

2016-02-08_system_log_api.png

4. Multithreading Flask

app.run(host='0.0.0.0', port=80, debug=True, threaded=True)

2016-02-09_threading_1.png

5. Using pgAdmin for Faster Query Generation 

2016-02-17_moon_5.png

6. Current State

There is a lot of polishing needed, but a lot of the ‘bricks’ as I call them have been put in place. Learning the basics of D3.js, how to return JSON from Pandas manipulated PostgreSQL data, and bootstrap-table for modern table UI all have been on a punch list to learn to get to this state.

2016-03-01_dashboard1.png

2016-03-01_deadend_1.png

2016-03-01_moons_1.png

2016-03-01_moons_2.png

2016-03-01_npc1.png

2016-03-01_nullsec_1.png

2016-03-01_sov_1.png

2016-03-01_tradehubs_1.png

2016-03-01_logs_1.png

What’s Next?

  • Bring on a partner that can help guide the frontend.
  • Deciding if Flask+Jinja2 is really the right frontend platform for this project (and me).
  • Migrating graphing to use height+length percentages rather than finite pixel amounts as the Bootstrap style supports many different formats. Currently the charts look horrible on any tablet or mobile device.
  • Parameterize charts and graphs.
  • Write consumer for CREST market prices.
  • Write consumer for zKillboard/CREST killmails.
  • Develop a proof of concept report for a Region that takes only the regionID as an input so that I learn how to build reporting for any region or Solar System.
  • Leveraging the power of Pandas by spending more time on the thinking problems: developing Indexes, finding trends, and learning about covariance.
  • Reach a good polish so I can publish a 0.3 release.
  • Learning how to create a Docker image.

Latest updates can be found on the 0.3-dev branch.

Advertisements

Vanguard Incursion Farming

For an excellent guide on the methods for running Incursions, review Jester’s Introduction to Incursions Part 1, 2, 3, and 4.

Given that I have Logistics V, Caldari Cruiser V, Shield/Energy Emission Systems V, and Multitasking IV, I found my way into an Incursion fleet running a Basilisk for a Shield fleet. I had Monday off and took the time to grind out Vanguard sites for stupid amounts of ISK. Our average was a little low due to members changing ships, but we still managed to pull in 93 M/hour.

My impressions of Incursions sites is that they are very formulaic. As a Logistics pilot, here are the steps to complete a Vanguard site in 4-6 minutes if you have the proper fleet composition:

  1. Get on voice comms. This is so you can talk about how easy Vanguard sites are and how rich you are becoming every 4-6 minutes. In actuality, you will use voice comms to yell at Logistic pilots if they are not paying attention or if someone deviated from the prescribed kill order and triggered the next wave.
  2. Anchor warps in to the site first which is usually your FC.
  3. Everyone else warps in, lands.
  4. Orbit the Anchor at 5,000 and lock up all your friends.
  5. Watch to see who got aggression first and start repping them.
  6. Before the site is finished, the FC will post an align to the next site. Active align.
  7. Site ends and rats despawn, fleetwarp to the next one.
  8. Go to step 2 and repeat.

I do agree that a change should be made to the risk/reward of Vanguard sites. Different waves, site types, and perhaps lengthening the time it takes for Vanguard sites seem reasonable in order to increase the amount of effort for the ISK.


POS Fuel Costs

I’m a creator at heart and now that we have gotten back into sovereign nullsec, I want to setup a research POS. Working with eve.1019.net’s POS planner, I setup a tower with six Advanced Mobile Laboratories to estimate fuel costs.

Two major events are increasing the cost to run a POS. GoonSwarm’s Gallente Ice Interdiction campaign has driven up prices for not just Oxygen Isotopes. All Isotope prices have gone up due to market speculation and general Mackinaw killing campaigns.

Gallente towers are used for moon harvesting due to their bonus 100% bonus to Silo Cargo Capacity values. Since I will not be moon mining, I picked a Caldari tower which uses Nitrogen Isotopes.

Additionally the announcement of Player Control Customs Offices in the Winter Expansion has caused all PI products to rise in price due to market speculation. Practically every PI item has been affected with precious Robotics taking a hard hit.

With Isotopes consuming a large amount of the ISK to run the tower, any ice mining that we do as a corp will help bring down the costs.

Faction towers do have lower fuel requirements than Racial ones. Putting the same 6 labs on a True Sansha Control Tower shows a 44.2 M/month savings on fuel.

 

Sadly the high price of the faction towers means that it will take 3.5 years to make up the cost difference using the lower prices True Sansha variant at 1.99 B ISK.


A Jump Freighter is Not a Toy

If you have ever read some of the shenanigans that myself and hav have gotten ourselves into while in WH space, you know that we are an entertaining bunch:

It seemed like a really good idea at the time

Today I moved a JF through a constellation that has an active Incursion going on. While an Incursion is active, rats at gates are more agressive and stronger than regular nullsec rats. Additionally with the Incursion that we have, you cannot use cynos. This meant that the very shiny, expensive ship had to go through the gate.

There was a miscommunication between myself and my scout James in the destination system. He thought that I was starting in the destination system. This meant that warping to 0 on the gate would get me past the Sansha rats.

I was in fact starting in another system and coming into the destination system that James was watching for me. He was watching three rats some 30k off of the gate and when I decloaked after the gateflash 15k away from the gate, James was quite confused.

[ 2011.06.20 22:43:13 ] (notify) Warping to ** prior to making docking attempt
[ 2011.06.20 22:43:19 ] (combat) Yulai Crus Cerebi hits you, doing 1434.5 damage.
[ 2011.06.20 22:43:24 ] (combat) Yulai Crus Cerebi misses you completely.
[ 2011.06.20 22:43:29 ] (combat) Yulai Crus Cerebi misses you completely.
[ 2011.06.20 22:44:47 ] (notify) Requested to dock at ** station
[ 2011.06.20 22:44:47 ] (notify) Your docking request has been accepted. Your ship will be towed into station.

[ 2011.06.20 22:42:04 ] Blake > warping to gate
[ 2011.06.20 22:42:09 ] Blake > hope this works
[ 2011.06.20 22:42:15 ] Raath > hehe
[ 2011.06.20 22:42:20 ] Raath > almost believed you there
[ 2011.06.20 22:43:21 ] Blake > f***
[ 2011.06.20 22:43:34 ] Blake > lol
[ 2011.06.20 22:43:36 ] Blake > LOL!
[ 2011.06.20 22:43:42 ] Blake > they webbed me and that sent me into warp!
[ 2011.06.20 22:43:43 ] Blake > LOL
[ 2011.06.20 22:43:52 ] James > How did you get so far out?
[ 2011.06.20 22:44:11 ] Blake > man that was the dumbest thing I’ve done yet
[ 2011.06.20 22:44:17 ] Blake > but funny
[ 2011.06.20 22:44:23 ] Blake > haavoc style funny
[ 2011.06.20 22:44:26 ] Blake > good thing they didn’t scram

Thankfully they were not scramming rats and instead made my Jump Freighter warp faster to the station because I was triple webbed.