MV3D Development Blog

I didn’t really go into detail too much as to how I solved load balancing and redundancy previously. In order to better flesh it out, I’m going to do so now. Please refer to the following diagram:

This is what I’m calling a cluster. It is what will be managing high availability in MV3D. Although there’s on big box there, the cluster is meant to run on as many CPUs as you can throw at it. I’ve just demonstrated a single node in the cluster for simplicity. At the core, it is a sharded pool. A sharded pool uses a plug in mechanism to link a given id with the pool it exists in. The default mechanism is a range based partitioning scheme. A basic example of this would be one that sends ids 1-1000 to pool A and 1001-2000 to pool B. That’s the general idea. Clusters use the default range mechanism to partition the ids into shards. These shards are themselves sharded pools. In this case, though, they use another mechanism that is optimized for a 1-1 mapping of id to pool. It is expected that the pools it maps to are redundant pools. That’s a whole new type. The redundant pool is as it says. All of its objects are duplicated across all the members of the pool.

So far, there’s a hole in the scheme. Where do these mystical ids come from? Anyone who has looked at MV3D knows that an id is either an integer (0) or a two integer tuple (0, 0) where the first number is the parent’s id and the second is the child’s id. For instance, in the case of an item, the first number is the realm id and the second is the item id within the realm. Anyway, these numbers come from an IDDispenser object. The cluster has one of those. It sits in a cluster-wide redundant pool. This means there is one master dispenser per cluster (and multiple redundant copies). This is where the ids come from. The redundant pool it sits in can also be used for other things that belong to the cluster. One particular use for this is to have an object which provides data for the pool. For example, an Asset Group would be a cluster, but there would be an AssetGroup object in the Redundant pool that has information about the group. To expand on that notion a bit further, the redundant pool for a Realm would contain the Realm object which contains physics properties.

I haven’t mentioned too much about the load balancing aspect of this though. Really, it’s fairly easy. The sharding of the top level cluster ensures that you can split up the management of the cluster across as many servers as needed. Then individual items in the cluster can be moved from pool to pool very easily to balance out the load. With directory, realm, and asset servers, the load balancing will be fairly manual. When you need more servers, add them to the appropriate cluster or pool. Load in these areas should be fairly predictable by standard means. The load balancing gets a lot more interesting when you start talking about simulated parts of the world.

Load balancing at that level is more tricky because the load isn’t predictable and is a lot more dynamic. 100 or more players could crowd into a shop where just moments before there were none. I’ll likely write about this in more detail with a future post, but the basic idea is to automate enough of the load balancing to make capacity planning predictable. MV3D will have a bank of servers partitioned to simulate a set of areas and a mechanism to balance that load across the servers. With that in place, capacity planning can be done by determining the load across the group of servers by changing which servers simulate a given area. Watching the increase over time should allow the operator to predict when more hardware is needed due to population growth. In general, all of your simulation servers would be in this group so that you can best make use of them. As I said, though, more on this later.