MV3D Development Blog

We have a WINNAR here. I did what I was talking about in my last post and made a way to keep track of values that changed often and to just automatically save them every once in a while. Basically, I made an Axiom Item for storing Vectors and Quaternions, then I created a new type of attribute for the MV3D datastore, which specifies that the value of that attribute is to be stored via an Axiom Item class. Since the only place that position/rotation is stored for objects is in the ODE body, I also made that attribute type able to use a getter/setter function to get/set the value to store.

Then from there, I added another option to them to specify that you want the attribute persisted automatically at regular intervals. If that’s on, it gets added to a list that the datastore keeps and every 15 seconds (currently), it stores the value again. It also automatically stores the value when the object it’s in is stored or when the store is closed. This seems to work great. It gave me a scare when I first got it working though–

Synced 480 autoStore objects in 19.34 seconds

Yeah, that was bad. But then I realized that I didn’t have it running in a transaction. I also added the smarts to it so that it doesn’t store a value if it hasn’t changed. That leads me to:

2008-05-31 00:09:43-0400 [-] Synced 6480 autoStore objects in 0.09 seconds

That was with not much changing.. Here’s two with about 800 objects all moving:

2008-05-31 00:00:04-0400 [-] Synced 9720 autoStore objects in 0.23 seconds
2008-05-31 00:00:17-0400 [-] Synced 9720 autoStore objects in 0.28 seconds

Those are all perfectly fine and don’t interrupt the server enough for it to be noticed (especially the 0.09 second value– the server only updates every 0.05 seconds).

There are a couple of bugs that I noticed, though. When the octree based area saves itself (which happens when something passes between nodes), it takes a while. Most likely, it’s saving way more data than it needs to, though. Also, (and this seems like it was a bug before I started messing with the datastores) after loading up a world, the physics is a little off from what the client expects. My theory is that there is some value for biped movement speed that isn’t being persisted or something isn’t being put in the right bucket when loaded.

All in all, though, I’m feeling pretty good about this.

Ok, so I made a good attempt at a store that allows you to queue objects up to be written to it asynchronously. There were a few tricky spots such as getting queries to use the correct version of the data. Basically, the store had a global revision that would get increased every time you called “sync”. The item in the store with the greatest id that was less than or equal to the store revision was considered to be the most accurate. After every sync, it would clean up old items.

Everything worked great except for the speed. Storing to an empty store took no time at all, but after that, even queuing an item to be stored started to take time. That meant that storing about 500 items took 2 seconds. Then the sync afterward took forever. The first time, it was about 40 seconds, then 120 other times. I’d be ok with the asynchronous part of things taking a bit longer, but 120 seconds was way too long. I’m scrapping this idea.

I started to think about the data I’m storing more closely though and have some ideas. Really, the only things that change very often are the physics attributes. Position, velocity, etc. I bet that I could store those directly to axiom pretty darn quickly. The fun part of those values is that they aren’t attributes of the objects that are being stored. They are attributes of the ODE objects. Currently, when storing them, a trigger gets called that plops them into the containing object as things like “initial_velocity”.

One thing I may try is making those things python properties that map to get/set on on the ODE object and then telling the datastore that they are values that rapidly change. The store could then add them to a big list of similar things that could be persisted at a separate time than the object as a whole (though persisting the object should also persist those values immediately). Also, I wonder if they should also be stored as a normal Axiom Item. That could make updating them easier.

So, this is my next thing to try. I’m feeling pretty positive about it, but I was feeling the same way about my last attempt as well.

So, I’ve migrated all of MV3D to using a new type of datastore that has Axiom as its back end. It’s pretty sweet and really wasn’t a lot of code. The interface is the same as the other MV3D datastores, so it was pretty much a drop in replacement. It is very fast and uses transactions for extra speed. I haven’t compared, but I think it’s slightly faster than my SyncFile datastore, which is what everything in MV3D used except the Sim service. The Sim service has the unique requirement that the data it needs to save changes 20x per second. In practical use situations, it’ll also have tons of data to store. Clearly, saving 20x per second isn’t an option, so instead, it saves everything every so often. One problem with doing that is that if you have 500 objects in an area, you have to save all 500 in the same physics iteration or else your simulation is out of whack when you load it. This makes for a pretty tough challenge, and I’d venture a guess that it’s one of the big reasons why MMORPGs don’t have completely dynamic worlds like MV3D.

The solution I came up with before Axiom was a memory based datastore that could sync to disk in its own thread. What would happen is that it would store all the objects in a big python dictionary in memory synchronously. With 700 objects, this takes a little less than 1/2 a second on my system. Then, it stores that dictionary to a flat file in a thread. This means that the simulation is only interrupted for 1/2 a second. That’s still not perfect, but I figured it may work to cheat a little bit and have say 1/2 of the objects be from t, and the other from t+1. That way, it’d only be 1/4 of a second hold up.

Unfortunately, I haven’t come up with a way to do something similar with the Axiom based store. Right now, it starts a big transaction and stores all the objects using Axiom which takes about 5 seconds. Having everything stopped for 5 seconds on the server is definitely not something I can do.

The best idea I’ve come up with so far is to grab all the attributes of the objects I need to save synchronously (which should take < 0.5 seconds) and then stuff them into a cooperator that stores a few at a time through Axiom. My only fear with that approach is that it’ll take a lot more time to store everything. Actually, this could be done in a nice way. I could make a new Axiom based datastore that stores things to memory initially, but when you call it’s ’sync’ method, it iterates through all the things you’ve stored and saves them through Axiom.

Of course, there is a relatively big issue with that approach. There is no way to do everything in a single transaction, so if something went wrong while it was half way through storing in Axiom, it can’t be rolled back. I could use a trick I learned from a DBA at my job, though. Add an extra attribute that signifies that an object in the database is active, and set that attribute on all the objects if everything succeeds. I’m not sure that would save me any time, but what I can do is to add an attribute for revision number (or timestamp) which would be the same for all objects and have a separate Axiom object that specifies the most recent revision that was successfully stored.