Cubeia Poker Cloud Scalability
Would you want some cloud with your game server?
Cubeia Firebase was initially build for real money gambling and for deployment on dedicated hardware. It makes sense to control your hardware when you’re dealing with monetary systems but as gambling focus has shifted the last couple of years towards social gaming cloud based deployments makes much more sense. In this article we’ll present Cubeia Poker running on Amazon AWS and our initial load tests and evaluation.
In order to make these tests as real-world as possible on this stage we made the following modifications to our standard suite of software that comprises Cubeia Poker:
- Cubeia Firebase – Normally Cubeia Firebase scales in all directions by using direct-connect inter-cluster communication. But as a cloud deployment usually cannot make hard guarantees on internal bandwidth and latency we settled for a pure scale-out architecture instead. This entailed disabling the internal message bus and state replication mechanisms.
- Cubeia Network – The main challenge here was to scale the wallet functionality. Again, a real money network have demands on consistency that can be relaxed in a social setting. Our modification will be discussed later, but in essence we added sharding of table transactions to scale the load.
- Cubeia Poker – A standard build was used except small changes to provide for the above changes to Cubeia Network. Note that the changes in Cubeia Firebase was completely outside the API and nothing needed to be changes in the Poker code to accommodate them.
The Cubeia Network accounting modules uses dual entry bookkeeping, with static accounts for the main balance of a player and session accounts for all transactions for a player at a particular tables. When a player signs up and plays at a table, this happens:
- The player signs up: a static account is created to hold the players balance.
- The player sits down at a table with X chips: A session account is created for the player and the buy-in sum X is transferred from the players static account to the session account.
- As player loose and win chips around the table the balances of the session accounts are updated.
- The player leaves the table: The session account is closed and the resulting balance (ie. win or loss) is transferred to his static account.
The code itself is write optimized and scales well on dedicated hardware. But for this test we opted to run the session accounts, ie table transactions, on separate servers and shard these across the cluster.
We opted to run the tests on Amazon AWS in their East Coast data centers, but paid no attention to exact location.
- EC2 was used for all Firebase, Wallet and Bot servers. Pre-setup AMI’s where used with live configuration via Puppet and user data scripts. Only ephemeral storage was used. All servers ran as High-CPU Extra Large Instances (c1.xlarge).
- RDS was used for the databases with MySQL 5 without and replication and read/write setup. Provisioned IO was not used. All instances ran as Extra Large instances with an allocated 5 GB of storage.
It should be noted that apart from the large instance types no effort was made to customize or otherwise tweak the AWS installation, the servers were not even in the same data center.
25 000 Concurrent Players
The Firebase and Bot servers where launched in pairs; one pairing of with one bot server with the bot server running 25 000 players. One Wallet server where handling the static account and one the session accounts. The CPU load on the Wallet servers where 25 and 20 percent respectively, and 12 and 18 percent for the matched databases (eyeball figures).
50 000 Concurrent Players
As the load on the Wallet server was perfectly acceptable the only addition for this step was one more pair of Firebase and Bot servers. The CPU load on the Wallet servers rose to 50 percent for both server, and 25 and 30 percent for the matched databases (eyeball figures).
75 000 Concurrent Players
One more Firebase and Bot pair was added. Another database shard and session account Wallet server where added together with an additional static account Wallet server to share the rising CPU load. The load on the Wallet servers matched the expected loads from 50′ + 25′ above.
100 000 Concurrent Players
For the final load a last Firebase and Bot pair of server where added. Again the load on the Wallet servers matched the expected loads from above, ie. 50′ + 50′.
Below follows some evaluation of the actual numbers pulled from the servers during this test. Please note that the game play frequency oscillates naturally and therefore small variations on load and relations are expected.
The above chart shows the executed game events in a Firebase server correlated with Poker hands per minute. At full load we were executing 4264 incoming events per second and generating a whooping 13 204 hands per minute (220 hands per second).
As expected from the previous graph the relations between executed events and hands per player stays flat. Each player generates approximately 2.4 events per minute and you can expect approximately 0.14 hands per minute and player.
Poker is a naturally “talkative” game. In Cubeia Poker each executed event generates approximately 10.5 outgoing events. At full load the system was executing 4264 events per second and broadcasting 46 264 events per second (approximately 1050 and 11 500 per Firebase and Bot server pair).
Using this we can calculate the cost per player to run this cluster. Two things should be taken into account here: 1) bandwidth is not included; and 2) no attempt was made to minimize the server cost by running the most effective instance type for each server. For example, each Firebase server had a CPU load of approximately 50-65% during the test and could possibly have been run on cheaper instance types. But apart from this the monthly server cost of running 100 000 concurrent poker players is $5378 which corresponds to $0.054 per player. The slight increase in cost at 75′ concurrent players is because the scaling model is not perfectly linear across this test, between 25′ and 50′ only one server was added, but between 50′ and 75′ four servers entered the cluster.
The total cost of running a Cubeia Poker cluster on Amazon is very low and decreases with more players despite the fact that Firebase is a heavy server (due to it’s real money origin) and that Poker is a very event heavy game. The scalability model works very well provided the limitations.
Ma’, can I go out and play with my one million friends?
In order to take this further the static account database needs to be sharded. Whereas session accounts deals with player to player interactions, static accounts do not, and the sharding is therefore relatively simple and can be done using user ID or any other available metric. One million? Sure, why not?
About the author: Lars J. Nilsson is a founder and Executive Vice President of Cubeia Ltd. He’s an autodidact programmer, former opera singer, and lunch time poet. Any questions are welcome at lars.j.nilsson in Cubeia’s domain.