MV3D Development Blog

It’s been quite some time since I’ve posted anything here about MV3D. Things have definitely been going slowly recently in that regard, though they are picking up again now. I work on an MMO all day, so when I get home, I generally want to do something else for a while. However, now that I’m more settled in, I’ve been able to start putting in the time again after work and on weekends.

Mostly, I’ve been trying to put together some content editing tools since the in game editor isn’t really the place where you’ll want to put all your content together. It’s good for terrain sculpting and adding foliage and such, but I think I’d pull my hair out trying to use it to make a town. One of the major methods of building complex things in MV3D is the modular object system (demo’d in a few of the videos I think). I’m working on a tool so that complex modular object systems can be designed without having to write code. It’s coming along really well. At this point, I’m not sure it really fits in with MV3D, so I’ve been keeping it on the side.

My goal is to get content creation and some semblance of an art pipeline up to a point where I can create a passable looking game world. I’m not expecting anything too wonderful because I have no talent making 3D models (and am therefore limited to GPL or CC licensed ones that I can find), but I should be able to come up with something. Then after that, I’m going to start working on putting together a demo game.

At this point in the development of MV3D, I’ve often been unsure what the next most important thing for me to work on is. If I start putting a game together, I’ll be able to quickly see what areas need help and work on those. At least that’s the theory.

One thing I really want to fix up is the client performance. Last time I ran a load test, having 50-100 characters running around caused the client to become fairly unresponsive. Not surprising since I’m not doing a ton to mitigate that.

I’m sure some people are probably hoping I’ll talk about 38 Studios and Copernicus. About all I can say is “Yes, it’s awesome, and no, I can’t talk about it.” Ok, I can say more than that. We’re hiring! Check out the 38 Studios Jobs Page. It’s really a ton of fun working with such an incredible group of people.

Hopefully, with things picking up on MV3D again, I’ll be posting a bit more. I’ve also started an MV3D page on Facebook. Mostly, I’ll use that for random related things that don’t really warrant a whole blog post. I’ll probably also put pictures up there that I wouldn’t want to put on the main page.

Seriously. I have XP 64 on my main system at home (ok, stop laughing) because I have 4gb of RAM and don’t want to make my Athlon X2 into a K6 by running Vista. Anyway, I’ve recently decided that it would be a good idea to move my music creation stuff from my old desktop to the new one. The old one just really can’t keep up any more and starts getting tons of lag when doing real time effects. Anyway:

There hasn’t been a lot of visible progress on MV3D recently. That’s mostly due to changes in my employment, but also because I’m working on a monolithic project to revamp the load balancing. I think I’m half way through, but I’m not really sure. Anyway, for the past 3 years, I’ve been the only developer working on MV3D. Mainly, I wanted to make sure that the foundation was in place and it had a clear vision. After these load balancing changes, though, I feel like the foundation will be there. So, I’m looking for help.

The big areas are client side programming, tools / editors, and help taming Ogre’s art pipeline. However, once the load balancing is done, I’m going to start working on Siegium again, but this time it’ll be as a demo world for MV3D that has game logic.

If you are interested, comment/email/post on the forums/send carrier pigeons.

I didn’t really go into detail too much as to how I solved load balancing and redundancy previously. In order to better flesh it out, I’m going to do so now. Please refer to the following diagram:

This is what I’m calling a cluster. It is what will be managing high availability in MV3D. Although there’s on big box there, the cluster is meant to run on as many CPUs as you can throw at it. I’ve just demonstrated a single node in the cluster for simplicity. At the core, it is a sharded pool. A sharded pool uses a plug in mechanism to link a given id with the pool it exists in. The default mechanism is a range based partitioning scheme. A basic example of this would be one that sends ids 1-1000 to pool A and 1001-2000 to pool B. That’s the general idea. Clusters use the default range mechanism to partition the ids into shards. These shards are themselves sharded pools. In this case, though, they use another mechanism that is optimized for a 1-1 mapping of id to pool. It is expected that the pools it maps to are redundant pools. That’s a whole new type. The redundant pool is as it says. All of its objects are duplicated across all the members of the pool.

So far, there’s a hole in the scheme. Where do these mystical ids come from? Anyone who has looked at MV3D knows that an id is either an integer (0) or a two integer tuple (0, 0) where the first number is the parent’s id and the second is the child’s id. For instance, in the case of an item, the first number is the realm id and the second is the item id within the realm. Anyway, these numbers come from an IDDispenser object. The cluster has one of those. It sits in a cluster-wide redundant pool. This means there is one master dispenser per cluster (and multiple redundant copies). This is where the ids come from. The redundant pool it sits in can also be used for other things that belong to the cluster. One particular use for this is to have an object which provides data for the pool. For example, an Asset Group would be a cluster, but there would be an AssetGroup object in the Redundant pool that has information about the group. To expand on that notion a bit further, the redundant pool for a Realm would contain the Realm object which contains physics properties.

I haven’t mentioned too much about the load balancing aspect of this though. Really, it’s fairly easy. The sharding of the top level cluster ensures that you can split up the management of the cluster across as many servers as needed. Then individual items in the cluster can be moved from pool to pool very easily to balance out the load. With directory, realm, and asset servers, the load balancing will be fairly manual. When you need more servers, add them to the appropriate cluster or pool. Load in these areas should be fairly predictable by standard means. The load balancing gets a lot more interesting when you start talking about simulated parts of the world.

Load balancing at that level is more tricky because the load isn’t predictable and is a lot more dynamic. 100 or more players could crowd into a shop where just moments before there were none. I’ll likely write about this in more detail with a future post, but the basic idea is to automate enough of the load balancing to make capacity planning predictable. MV3D will have a bank of servers partitioned to simulate a set of areas and a mechanism to balance that load across the servers. With that in place, capacity planning can be done by determining the load across the group of servers by changing which servers simulate a given area. Watching the increase over time should allow the operator to predict when more hardware is needed due to population growth. In general, all of your simulation servers would be in this group so that you can best make use of them. As I said, though, more on this later.

After about a week of banging my head against the wall trying to figure out a better way to do redundancy in MV3D, I think I’ve finally got it worked out. I even have code to back it up. What I’ve created is a redundant pool of objects. Servers can join or leave the pool, and objects are replicated to all servers in the pool. One server is the master and the rest are slaves. If the master goes down, one of the slaves picks up the slack. The master can re-join the pool as a slave. It seems to work great.

The next step is writing a million more tests to make sure it acts properly in every situation I can think of. Once that is done, I think I can build on the platform I created to make different types of HA pools. For instance, one that divides objects up into sub pools.

Anyway, I’m pretty happy with this new method of keeping things redundant. It’ll take a while to integrate it into the rest of the code, but I don’t foresee any major problems. It is pretty cool though, I just implemented a method to move items from one pool to another. This could be used when changing the partitioning around, or (if I do one pool per area) when an item moves from one area to another.

I’ve been working on ticket #211 for MV3D lately. If you looked at the ticket, you’ll see it’s something that’s been kicked around for a bit. The current redundancy and load balancing code is fairly broken in trunk. It did work at some point in the past, but a myriad of changes have caused it to break in various ways. Clearly, one problem is that there weren’t enough unit tests for it or this would never have happened. The larger problem is that the code was very repetitive and not very organized. I’m setting out to fix this, but it’s not been easy. I’ve written a bunch of code and tests, but I have no idea if it is remotely the right direction to go in. Not to mention that this is the 3rd or 4th major direction change I’ve had since I started on this ticket. Nonetheless, I really want to get these details worked out before MV3D gets much further along. The longer you wait on something like this, the harder it gets to fix in my experience.

If what I’m talking about makes no sense, read up on MV3D’s Server Architecture. The high level requirements are that there be no single point of failure for any service MV3D provides and that servers below the Directory servers can come and go with little disruption. Starting at the top, a Directory should be split horizontally (by item id) across a number of servers. Each part of the Directory should be replicated to several servers. Asset Groups and Realms should get the same treatment. Realms should organically divide items across Simulation Services based on the Area they are in. All Items should exist on at least two Simulation Services. To further complicate things, some types of Areas can spread themselves across multiple servers, which means the objects within them have to do the same. It is very important to keep objects in the same Area (or piece of an area) on the same server so that there is no time lag for physics collisions.

The current way of doing this makes each level in charge of distributing the load for sub-levels. This means that there is also no method to get HA Directory Services. The Directory stores a master and a list of slaves for each Realm and Asset Group. Asset Groups also have no redundancy right now and must exist on a single server. The Directory Service is able to promote and demote copies of Realms on various servers from master to slave. Realms manage a list of Areas and what servers they can be simulated on. Each Area manages where its objects are simulated. This generally makes for a big mess without much rhyme or reason as to where anything is served from. Another problem I’ve encountered is that having all HA items be pb.Cacheable makes it hard to switch them from slave to master or back without re-caching them (which would cause confusion to any object that kept a reference to the old version).

I was going to write about my solution, but after I started, it became clear that it was completely wrong, so back to the drawing board.

I’m finally gearing up to make another official release of MV3D. This release includes quite a lot of new content creation features including:

I’ve actually had some time to work on MV3D lately, which I’ve been happy about. I did make a short video of the in game editor. It’s nothing wonderful (though I really like the music I came up with for it).

One of the things I did over the long weekend was to start making the client behave a little more like a client and less like a dumb terminal for the server. Previously, when you connected to a server, it would just shove all of the objects in your view range at you. Now, the client gets to pick. This has the added bonus that the client loads the images from closest to farthest. It can also limit the view range to less than the maximum to save bandwidth or cpu. The client is still limited to getting items which are in its view range by the server of course.

The other big thing I did– I’m still amazed I got this working– was to completely remove the need for the huge Media directory for the client. It’s about 65MB. I had to keep a few things. Just the GUI related files and the background image. Everything else is downloaded through the resource system in game like it is supposed to be. This should reduce the initial download size for the client by a lot.

That brings me to the next thing I’m working on, which is getting a Windows installer for the Client to work. I’m trying nullsoft to see how that goes. It seems pretty simple. I’d like to make some sort of script to put together the release using py2exe and such. Currently, there’s a bunch of manual work involved after the py2exe step.

Currently, there are 18 tickets left in the release. Hopefully I can get those finished up this month and get a new release out. That’ll depend on if I can keep feature creep out of there. I really want to make a simple “Add item from template” type deal in the editor since right now adding an item is rather useless.

First off, websites are hardly ever “finished” as I’m sure many of you know. Anyway, I’ve got the site up for real now although it’s still lacking in content (Corey, put some stuff up there, hint hint). Working with Nevow and Axiom was great and enabled me to build the site in a very short amount of time. The brunt of the work was done in about a month. It’s got a complete admin interface to it including image uploading. I also wrote a full featured shopping cart for the first time. Check it out:

http://www.cescenics.com

By the way, don’t those calendars look nice? I know you want to buy one.

Another cool thing– got to give some credit to PayPal. Their sandbox is a great tool, but also they’ve added some really neat stuff since I last used their API. You can have them notify you via a hitting a URL of your choosing when the status of a transaction changes. So, I’ve used that to verify that a purchase has actually gone through.

This website plus lots of extra hours at my job have really slowed down MV3D work. I’m currently finishing up the in game ui for adding and removing objects. Unfortunately, for every in game editor piece I finish, I find about 5 more that need to be done. One I just thought of today are body modifiers– so you can modify the physics properties including colliders and also the visual properties like models, textures, scaling and placement.

I have found a new web host though. And I’ll be attempting to move trac over there shortly.

I’ve heard a good amount of complaints made against Nevow, and one of them was that templates can’t include other templates. I’ve been working on a web based resource management system for MV3D this weekend and of course, it uses Nevow. I decided it would be nice to have a main layout template that included other templates for navagation and the content of the page. But that can’t be done with Nevow, right? Wrong. It’s pretty easy to include other templates. Just use rend.Fragment– it even suggests using it that way in its docstring.

Before I get in to specifics, I’d like to just comment on what I just said. Another complaint that I’ve heard about Nevow is that there is barely any documentation. In some cases, I can’t disagree– if you are talking about missing docstrings or an extensive manual. One thing I’ve learned, though, is that if you want to know how to use something, just read the code. The only Divmod code I’ve found so far that was hard to read was deep inside Axiom. Definitely part of that was because I didn’t really understand metaclasses at the time. What I’m saying is that I’ve always found the quickest way to answer a question is to just get out the code.

Anyway, rend.Fragment isn’t mentioned anywhere in the Nevow getting started manual that I know, but it took me 20 seconds of skimming rend.py to find it and even before reading the docstring, I knew it was what I was looking for. I also found that there’s a nice example of how to use it in exactly the manner I wanted in the examples folder.

What’s neat is that you can use it to cache chunks of your pages really easily. Say you have a header bar on the top of all your pages that is always the same for users who aren’t authenticated. Just make a subclass of rend.Fragment to render it and use the same instance of that everywhere. If that header changes for logged in users, then you can easily switch which rend.Fragment subclass you display in the header section depending on whether the user is logged in or not.

I could give an example usage here, but the best place to look is in Nevow/examples/fragments/.

In other news, there was something strange going on with the MV3D alpha server where it wouldn’t allow people to log in after creating an account. I’m not sure what the deal is, but somehow, I’ve fixed it. So, please give the new release a try if you haven’t already.