MV3D Development Blog

I just released MV3D 0.32 today. It has all the features I mentioned in a previous post plus now a load tester. There’s a slight issue with the load tester, though. It can’t handle load very well. The load tester, not MV3D. I had it going with about 75 clients connected and moving around, but the load tester was going very slowly. The problem is that currently, the client has no choice on which objects it gets updates for. The server decides which objects are within view range and automatically makes sure those exist on the client. Since the load tester impersonates any number of clients (75 in the example above), that means 75 player objects get updated 75 times. I have a ticket for the next release that will stop this behavior and allow the client to choose which objects to receive updates on (of course limited to ones the server deems are in its view range). I suspect that will fix the load tester and allow it to test more effectively.

However, 75 clients connected is pretty impressive. I was even able to log in via a real graphical client and walk around a bit. It got choppy now and then, but for the most part, other than a low framerate (30-50ish), it was pretty responsive. Impressive considering there were three apps running on my system that wanted 100% of a single core, and I only have 2 cores. I didn’t end up making any changes based on the load testing I did. That was mostly due to not being able to stress the server enough to make it matter.

My next task is to figure out what the 0.34 and 0.36 releases mean. Now that I have persistence solved for the moment, there are two areas that need attention. One is the world editor, and the other is load balancing and redundancy. My current thinking is that since load balancing and redundancy worked previously and the framework for them is still in place, it should be ok to focus on world building first. I figure that there won’t be any need for load balancing if no one can make worlds other than the test world. In any case, it’s time to do the ticket shuffle.

It took a bit longer than expected, but MV3D version 0.32 is almost ready to go out. At this time, I’m looking for people to test and report any bugs you may find. I’ve upgraded the alpha server (alpha.mv3d.com) to this version and have it up and running now. You can grab the client and server from SVN if you’d like. Note that a new Media package is available in the downloads section of the site. If you are running Windows and just want to try out the client, I have a zip file with everything you need (including media) located here:

http://download.mv3d.com/MV3D-0.32-rc2-WindowsClient.zip

You may also require:

http://download.mv3d.com/vcredist_x86.exe (Visual C++ Redistributable)
http://download.mv3d.com/dxwebsetup.exe (Direct X)

Before logging in to the alpha server, you will have to create yourself an account using the login server:

https://login.mv3d.com:8080/

I hope to wrap up this release soon.

New features in this release:

• Graphics
  • Trees
  • Grass
  • Clouds
  • Multiple player models
• Basis for the new in game editor
• Server
  • Axiom based persistence engine
  • Proper logging
  • SSH REPL for admin/debugging
• In Game
  • Character generator including model selection
  • Chat works
  • Enhanced biped physics
  • Working water physics

Here’s a couple of screenshots:

We have a WINNAR here. I did what I was talking about in my last post and made a way to keep track of values that changed often and to just automatically save them every once in a while. Basically, I made an Axiom Item for storing Vectors and Quaternions, then I created a new type of attribute for the MV3D datastore, which specifies that the value of that attribute is to be stored via an Axiom Item class. Since the only place that position/rotation is stored for objects is in the ODE body, I also made that attribute type able to use a getter/setter function to get/set the value to store.

Then from there, I added another option to them to specify that you want the attribute persisted automatically at regular intervals. If that’s on, it gets added to a list that the datastore keeps and every 15 seconds (currently), it stores the value again. It also automatically stores the value when the object it’s in is stored or when the store is closed. This seems to work great. It gave me a scare when I first got it working though–

Synced 480 autoStore objects in 19.34 seconds

Yeah, that was bad. But then I realized that I didn’t have it running in a transaction. I also added the smarts to it so that it doesn’t store a value if it hasn’t changed. That leads me to:

2008-05-31 00:09:43-0400 [-] Synced 6480 autoStore objects in 0.09 seconds

That was with not much changing.. Here’s two with about 800 objects all moving:

2008-05-31 00:00:04-0400 [-] Synced 9720 autoStore objects in 0.23 seconds
2008-05-31 00:00:17-0400 [-] Synced 9720 autoStore objects in 0.28 seconds

Those are all perfectly fine and don’t interrupt the server enough for it to be noticed (especially the 0.09 second value– the server only updates every 0.05 seconds).

There are a couple of bugs that I noticed, though. When the octree based area saves itself (which happens when something passes between nodes), it takes a while. Most likely, it’s saving way more data than it needs to, though. Also, (and this seems like it was a bug before I started messing with the datastores) after loading up a world, the physics is a little off from what the client expects. My theory is that there is some value for biped movement speed that isn’t being persisted or something isn’t being put in the right bucket when loaded.

All in all, though, I’m feeling pretty good about this.

Ok, so I made a good attempt at a store that allows you to queue objects up to be written to it asynchronously. There were a few tricky spots such as getting queries to use the correct version of the data. Basically, the store had a global revision that would get increased every time you called “sync”. The item in the store with the greatest id that was less than or equal to the store revision was considered to be the most accurate. After every sync, it would clean up old items.

Everything worked great except for the speed. Storing to an empty store took no time at all, but after that, even queuing an item to be stored started to take time. That meant that storing about 500 items took 2 seconds. Then the sync afterward took forever. The first time, it was about 40 seconds, then 120 other times. I’d be ok with the asynchronous part of things taking a bit longer, but 120 seconds was way too long. I’m scrapping this idea.

I started to think about the data I’m storing more closely though and have some ideas. Really, the only things that change very often are the physics attributes. Position, velocity, etc. I bet that I could store those directly to axiom pretty darn quickly. The fun part of those values is that they aren’t attributes of the objects that are being stored. They are attributes of the ODE objects. Currently, when storing them, a trigger gets called that plops them into the containing object as things like “initial_velocity”.

One thing I may try is making those things python properties that map to get/set on on the ODE object and then telling the datastore that they are values that rapidly change. The store could then add them to a big list of similar things that could be persisted at a separate time than the object as a whole (though persisting the object should also persist those values immediately). Also, I wonder if they should also be stored as a normal Axiom Item. That could make updating them easier.

So, this is my next thing to try. I’m feeling pretty positive about it, but I was feeling the same way about my last attempt as well.

So, I’ve migrated all of MV3D to using a new type of datastore that has Axiom as its back end. It’s pretty sweet and really wasn’t a lot of code. The interface is the same as the other MV3D datastores, so it was pretty much a drop in replacement. It is very fast and uses transactions for extra speed. I haven’t compared, but I think it’s slightly faster than my SyncFile datastore, which is what everything in MV3D used except the Sim service. The Sim service has the unique requirement that the data it needs to save changes 20x per second. In practical use situations, it’ll also have tons of data to store. Clearly, saving 20x per second isn’t an option, so instead, it saves everything every so often. One problem with doing that is that if you have 500 objects in an area, you have to save all 500 in the same physics iteration or else your simulation is out of whack when you load it. This makes for a pretty tough challenge, and I’d venture a guess that it’s one of the big reasons why MMORPGs don’t have completely dynamic worlds like MV3D.

The solution I came up with before Axiom was a memory based datastore that could sync to disk in its own thread. What would happen is that it would store all the objects in a big python dictionary in memory synchronously. With 700 objects, this takes a little less than 1/2 a second on my system. Then, it stores that dictionary to a flat file in a thread. This means that the simulation is only interrupted for 1/2 a second. That’s still not perfect, but I figured it may work to cheat a little bit and have say 1/2 of the objects be from t, and the other from t+1. That way, it’d only be 1/4 of a second hold up.

Unfortunately, I haven’t come up with a way to do something similar with the Axiom based store. Right now, it starts a big transaction and stores all the objects using Axiom which takes about 5 seconds. Having everything stopped for 5 seconds on the server is definitely not something I can do.

The best idea I’ve come up with so far is to grab all the attributes of the objects I need to save synchronously (which should take < 0.5 seconds) and then stuff them into a cooperator that stores a few at a time through Axiom. My only fear with that approach is that it’ll take a lot more time to store everything. Actually, this could be done in a nice way. I could make a new Axiom based datastore that stores things to memory initially, but when you call it’s ’sync’ method, it iterates through all the things you’ve stored and saves them through Axiom.

Of course, there is a relatively big issue with that approach. There is no way to do everything in a single transaction, so if something went wrong while it was half way through storing in Axiom, it can’t be rolled back. I could use a trick I learned from a DBA at my job, though. Add an extra attribute that signifies that an object in the database is active, and set that attribute on all the objects if everything succeeds. I’m not sure that would save me any time, but what I can do is to add an attribute for revision number (or timestamp) which would be the same for all objects and have a separate Axiom object that specifies the most recent revision that was successfully stored.

It’s really odd to be just heading to bed when my wife’s alarm clock is going off. She’s had early morning schedules and I’ve had late nights trying to just get everything done.

I’ve been making slow progress with MV3D, but still progress. One person has been helping me out with the visual side of things, which is great. He’s gotten a few PC models in place that are borrowed from Ogre. One of them is a dragon model which I’d like to make fly around (instead of walk) because it’d just be helpful in testing to be able to fly around the map. I’ve added a very simple character generator that now allows you to select which PC model you want to use. So, next release, I’ll set up the demo server with no PCs on it and let people create their own account/pc.

I’ve done a few other minor tweaks and stuff, but what I really want to ramble about is the new world editor. I have a few vague ideas, but nothing too concrete. I know that I want it to be completely plug-in based. It also needs to adhere to the MV3D security model, which I don’t believe the old one did. Mostly I’m thinking that it may not offer the user the same set of plugins depending on the area/realm/server they’re on.

That may sound strange, but when’s the last time you needed to adjust terrain– in space. It also has to be fairly security conscious, so sim services may want to have a list of editor plugins that are allowed. Maybe realms/areas/objects/bodies provide a lists of editor plugins that work on the thing.

So, a body may have move, scale, modify material. An object would have name, stats, add/remove body. Areas would have add/remove/select object/link (to another area). Realms would have settings for weather, add/remove/select area.

It would be nice if the different services running could add types to that. So, running a realm service would give you the realm editor, running a sim service would give you area/object/body editor. NPC services could let users add and edit NPCs in game.

Maybe this should be broader in scope than a world editor. Maybe it should really be a server control panel type thing. Don’t know. Have to think more about it, but I want to get this started.

Though, now, it is time for sleep.

I’ve been working on this for quite some time, and I decided that the 0.3 release was a good time to go back and finally finish it up. Here’s a video of MV3D:

Note that it shows features that are not in V0.3, but are in trunk (or have been temporarily disabled like the in game editor).

So, I was browsing around on some forums for an MMORPG “construction kit” and saw someone from that company making a post about how they currently don’t support collision detection at all on the server, and they don’t have any plans for it other than some very basic ability for game designers to check if two bounding boxes are intersecting. That wasn’t planned until at least two releases out, and this product is >1.0. Anyway, from what the guy said, it sounded like they did all their collision detection on the client. What an amazing idea! Surely no one would ever want to make a client hack to let you walk through walls. Inconceivable! Just makes me wonder what other decisions they’ve passed off to the client.

I’m not sure if I ever mentioned it here, but Second Life does physics / collisions similar to how MV3D does. The physics engine runs on the server (though possibly not on the client, which accounts for some amount of chunkiness, I’d guess).

In other news, here’s a screenshot of trees that I’ve put back into MV3D. This time in a non hacky manner, and you can’t walk through them on the client OR the server.

Up next is putting grass back in. Hopefully it won’t slow everything down this time.

beat me to it, but check out MV3D’s website (or this url if your DNS server is slow).

I spent the last few days getting Trac upgraded and installing a bunch of plugins. I also did some CSS customizing. Then I installed phpBB and decided not to bring over the old message board content. It’s all really out of date at this point, so better to start anew. And ergh, now I have to go and mess with the phpBB templates to make it match the rest of the site. Not looking forward to that (I spent way too much time on the Trac CSS).

Anyway, it’s official, MV3D Open Source Release, version 0.30. Get it while it’s hot. Or something.

I’m just about ready for the first open source MV3D release. I have a few decisions to make about the website, then I think it would be helpful to have up to date documentation that at least gets people started with the server and client.

There’s still one bug ticket left for the release, but I haven’t been able to reproduce the issue, so I’ll probably make the release without fixing it. That’s not to say that there aren’t any other bugs. I’m sure there are tons! And to repeat myself, there’s not even the remotest hint of a game here. This is just the simulation framework.

While I was doing some testing this weekend, I had up to 900 terrain chunks and 2000 cubes going. One terrain chunk is 1000×1000 meters(ish). Suffice it to say, I didn’t bother walking from one side to the other since it would have taken me too long. Performance is pretty good. Building terrains on the client side slows things down a bit. I’m debating between using libnoise or using a cooperator to spread out the work a bit. By the way, with 900 terrain chunks, the server was clocking in at just under 1gb of RAM use on Windows.

One of the great improvements in the last couple of weeks is that objects on the client wake up immediately as soon as they need to now. Before, they wouldn’t wake up until the next time the server considered them (which with 900 terrain objects and 2000 cubes could be a while since it only thinks about maybe 20 non-active items per second).

I did have to track down an annoying invalid pointer reference today. Something in Ogre was storing a regular pointer instead of a smart pointer to a mesh, so when the mesh was destroyed, that pointer was invalid.

I’m still debating about pointing www.mv3d.com to trac.mv3d.com. And I just realized I still need to open some ports to allow the login server to be reachable outside my firewall.

That’s all for now. Hopefully a release will happen this week!