The EveDataCore.com project was commenced in 2008 as a small Database driven Eve-Online website. It quickly outgrew the confines of a single database with the static database and became a multi-database project with several clusters.
This is a case study for how the project came to be where it is today, and the lessons which were learnt during it's conception, development and deployment.
In the beginning there was an SDE (Static Database Export)
CCP Games saw it in their generousity to release a database extract from their game environment for the digestion of developers. We took that database and initially started to build a website displaying the in-game items and marrying that to calculations which were applied in-game.
EVE Online has several hundred items in-game, each of which have their own characteristics and attributes. We took those attributes and began building an interface to display this meaningful data.
Then there was an API
An XML powered API was introduced not long after EveDataCore.com came into existence and was used to integrate player data and the Static Data together. It was at this point that a meaningful CMS was required and some concerted effort taken to design and develop a custom integration.
The new integration leveraged the Joomla Core and married the CCP API, SDE and Joomla CMS together into a very impressive application.
EveDataCore began consuming PVP and Transactional data shortly thereafter which also contributed to the increasing database size and CPU / memory requirements to process, store, and retrieve this data. This was where the project was required to migrate from a small VPS to a dedicated server.
Outgrowing the VPS and migrating to a Dedi.
The initial incarnation of EveDataCore was hosted with a shared hosting provider until such time as elevated permissions and custom scripts were required. When the custom database and application development commenced the site moved to a VPS with some generous hardware and bandwidth allowances.
With the subsequent growth of the database and the processing requirement the VPS allocation was quickly outstripped and a Dedicated Server stood up with another hosting provider. This is where the site has resided since and has been developed on top of with new database technology and infrastructure.
Clustered Databasing and Load Balancing
Now that EveDataCore was operating on a dedicated server providing the Joomla CMS and computational capabilities the database layer required attention. This new implementation meant an SQL cluster was required and the subsequent load balancer to share the read/write operations.
In the beginning we implemented a pure MySQL Replication approach, but it was not stable enough to maintain integrity. Now the EveDataCore is operating a triple MySQL node with a HAProxy load balancer facing the public side. This allows the Load Balancer to distribute the read / write operations amongst the nodes, it also means larger more complex queries can be distributed across the cluster.
Where to from here?
The EveDataCore will continue to be developed and enhanced over the continuing life the project. McHughTech will be contributing to it's development on a regular basis and developing innovative functions.