MV3D Development Blog

Not yet.

We have the first official MV3D Game server running on what will be the alpha server (NightEyes). Unfortunately, there isn’t much to do on it yet. In fact, according to my specs on the initial Alpha Test, there won’t be much to do on the server at that point either. However, I will say that with the exception of terrain, the new version of MV3D is a lot further along than the old version. Go Python. I guess the old version could download assets automatically by http, but IIRC, that would crash your client about 6/10 times.

NightEyes will have a twin server probably called Nosy soon. Nosy will be the development and internal testing server while NightEyes will host the (hopefully) always running alpha/beta game instance.

What can you do on the server? Well, in the Linux client (there is no Windows client right now due to the bug in Ogre that still causes time to stop), you can connect (there is only one PC so far, so if multiple people log in, they get the same view– though yes, I have tested that). You can fly around and watch the robots wobble on the floor because the physics engine isn’t yet disabling objects that aren’t moving. Since real time physics engines do not simulate certain types of friction and stuff, objects have a tendency to wobble after they fall to the ground instead of completely settling down. Oh yeah, you can headbutt the robots and make them fall over. I have yet to get the domino effect on them since when you headbutt them, your view goes twirling off into space, so it’s hard to hit things just right.

The speed is pretty good. I haven’t tried with more than two clients logged in. The client doesn’t do any physics, so the objects only get updated when the server sends new positions along.

Meanwhile, I did run the server with up to 5000 objects on my desktop computer (P4 2.6ghz) and got 10 iterations per second with the loop that updates object positions for clients turned off. Unfortunately, with it partly turned on, 1000 objects got the same performance. And fully turned on, only 500 objects. I have developed some optimizations that will hopefully fix this. Especially considering that the server really shouldn’t be thinking about any objects that aren’t moving at all and the majority of objects won’t be moving at the same time. Just players, NPCs, and anything that they are currently interacting with.

I still have to get my asynchronous database code working. It currently saves ok but doesn’t load because you need to give it info about every category of object you may have it load before loading them. So I’m working on a way to fix that. (It requires a constructor for each class for those in the know) I actually just finished the testing on the fix, but haven’t integrated it into the database code at all.

I came up with a description of the features that need to be in the alpha test for it to be considered ready:

1.1. Server

• Stable. The server must not crash often and be able to run continuously for long periods of time. So far so good

• Secure.
  • All connections must be required to authenticate themselves before getting any access. Done

  • Individual objects should have permissions that can restrict what specific accounts or groups can do related to them. Done
  • Clients should only be given views of the gameworld that are limited in scope to what their PCs can see. Done
• Persistant. The world should be stored in a long term manner. The server should be able to save and load all objects required to reconstruct the game instance. The game state should be automatically saved at a certain interval. Getting there
• Physics.
  • Objects should have realistic collision responses when desired. Done

  • Players should walk on the ground.
• Chat. Players should be able to chat with each other.
• Editing. Basic God-Mode in game editing.

1.2. Client

• Login.
  • Clients should be able to select a MV3D server and log in to it.

  • They should be able to create basic PCs (not many options req’d)
  • Clients can connect to a given PC
• Physics. Client side Physics should be enabled.
• Assets. Assets should be downloaded automatically as needed.
• Graphics. Just the basics. No terrain required
• Sound. None required. Done

PSKudos to anyone other than who can guess where my server naming scheme comes from. I have other servers with the same naming scheme if you need clues.

.. Cause I do.

See:

My robot army obeys the laws of gravity and falls into the black abyss below it. Pretty cool. It’s supposed to do that at this point, and that’s really all it does. The good news is that the robots exist on the server, and the client only gets to view a few things about them (what they look like, their names, and where they are). The server was having a ball keeping up with it though for some reason. You’d think the client that was rendering hundreds of 3D images per second would suck up the CPU. I’d think that too. But we’re both wrong, so there!

After much pain and a lot of digging through Other People’s Code, I also have finally gotten Asyncronous Database Saving/loading working. Turned out the problem I’d been having all week was that threads don’t work when they are run from the code that gets executed when you import a module. Yeah, I don’t understand what that means either. Oh yeah, MySQL 5 doesn’t work with Python.

I’m sure everyone is wondering what happend to my computer. I pulled the OS hard drive (which had Linux and Windows) and reinstalled Windows then Linux on a spare one. Works pretty good now, though I have some Video work to do which means I need to reinstall all my video software.
The next step (after the required video work) will be to give the robots a ground to fall on and see why the server was going so slow. I may also look into Psyco since that could help speed up the server. After they have a ground, I should look into figuring out how to set realm-wide things on the client– like gravity. Oh and the client should get some notion of physics as well. At some point, I need to learn how to sense when someone hits ESC when running the client cause right now, you need to Alt+Tab and then Ctrl+C.

Just a quick note. USPS lost two of the videos I sent out from the last job I did. WTF?

No updates for a while probably.. I was trying to compile PyOgre (the Python Ogre module) on Linux and kept running out of RAM. I only had 512mb because my motherboard used rambus memory and it is prohibitively expensive to buy. Instead, I bought a new motherboard and CPU fan for about 1/2 the price of getting another 512mb of Rambus memory. Rambus=teh sux0rz. Anyway, installed the motherboard last night and Windows XP refuses to even attempt to boot (bluescreen before seeing anything else). Linux pretty much works fine, but it has decided that my ATARAID controller card is the devil. Perhaps that’s Windows problem too since Linux does an Oops* or something when trying to mount a drive off the RAID controller.

So… That plus the fact that I can’t upgrade my Linux kernel to v2.6 because of the RAID controller is making me reconsider using it. It’s crap anyway because the hardware is just an IDE controller and the drivers for it do all the RAID in software. So I’m probably better off doing software RAID in linux/windows. Just I think they don’t play nice with each other if you want to use one RAID array that has both Windows and Linux partitions on it (like I’m doing now). I think I’ve come to the conclusion that I don’t need RAID in Linux. I don’t store anything useful on my Linux box really. All the source code lives in the SVN repository on another system (which uses a similar ATARAID card, but doesn’t have Windows on it or any need to be upgraded to 2.6 kernels).

I’m considering rebuilding both my Windows and Linux installs on my computer since they are both screwed up in other ways as well. Since I wanted to keep my previous Windows 2000 install until I was finished upgrading and was sure it would work, my Windows XP install got stuck on what it calls the E: drive. And I have a C: drive that has Win2k on it which I never use and can’t get rid of because WinXP thinks that’ll cause the end of the world. (It probably will because it also is my boot drive with LILO on it, but Windows doesn’t know that)

Hopefully I can use the RAID controller as an IDE card because I do need to keep RAID in Windows, and the onboard ID supports 4 drives (DVDR, Boot Drive, and 2 large drives for my video production biz).. In Linux, I installed VMWare tools when I was running that installation as a VMWare client OS on Windows. Now it screws up my X settings every time I boot (it didn’t do this a few days ago which is odd since I haven’t used VMWare on it in years).

That’s all for now other than almost buying a new shiny car the other day. Stupid car dealers. They suck.

* Oops: Seriously, linux Kernel errors are funny. It really says oops. One of my favorites (which I have not personally seen) is “LP1 on fire!” (as in “the printer, the printer, the printer’s on fire! We don’t need no water….”)

Ogre may not be such a sure thing. I took the first Ogre tutorial and put in some simple MV3D code that uses Twisted to connect to a MV3D server. Unfortunately, it seems like Ogre (or PyOgre) messes with the system timer causing time to basically stand still for the Ogre application. Twisted was very confused by this and kept saying the connection timed out immediately. Using Twisted to schedule a function to be run in say 5 seconds caused that function to run immediately. So it pretty much makes Twisted useless. This is no fault of Twisted’s because if I use Python’s time and date modules while running an Ogre app, they return the same time. I posted to the PyOgre forums about it, so we’ll see what happens. If no one has a clue or it’s a bug that can’t be easily fixed, then Crystal Space it is. I also want to see if the problem happens in Linux since I suspect it doesn’t. Other people are using PyOgre+Twisted, so I have to assume either it works in Linux, or something is borked with my Windows setup other than Windows being borked in the first place.

Speaking of borked though, that also describes my Linux partition on my desktop. Well, it was mostly working last night, but Debian was telling me that I basically had to do a dist-upgrade in order to install any new packages, so I’m doing a dist-upgrade now.

Aside from the time problem, Ogre does not have a suitable Terrain system nor does it have portals at all. :-/ Actually, I think the major difference between it and Crystal Space is that Ogre has a lot less features, but they are better done. Crystal Space has portals of course and a few Terrain types to choose from. I want the good parts of them both, dammit! Ogre also has pretty good documentation. With Crystal Space, it’s “Isn’t the code documentation enough? I mean, there’s almost half a million lines of it for you to read through!”

Well, maybe this is someone trying to tell me to finish up more Server related code like auto-perist to database and load balancing of already loaded areas (load balancing of new areas is in place already).

I did some more playing with Ogre today including downloading the demos. They were about 35MB, which seemed odd to me since the bare essentials of Crystal Space is 200MB (which compresses to about 50MB in a zip file). The demos were very good and showed off quite a few features that Crystal Space just doesn’t have. I went on to download the Python development environment for Ogre. Once again, I was confused as these were only 5MB. I figured I must have to download the rest of Ogre to use them (sort of like Crystal Space’s Python implimentation). Nope. 5MB and that’s it (plus 12mb for some demo data like models and such that was a seperate download). I installed it (and the prereqs– python 2.4, and psyco) and after having to download a random DirectX DLL, I was able to run the demo apps. Last time I got Crystal Space working on Windows, it took me about a week. This was an hour tops. So I think pending an investigation on whether Ogre supports some required things like Portals (which it claims to), I’ll probably be using it for MV3D.

That said, clearly, I’ve been working on client stuff today. Just for kicks, I tried running a server on Windows and after some major tweaks to my Date class, it works. Dates are still way screwed up, but I can run the unit tests (except MySQL related ones since I didn’t install the Python MySQL module– the client won’t use that).

I figured out what sort of view clients are going to have on things finally as well. I’d been thinking that I would just give them a cache of the object they control and then have some sort of interface for them to send commands and request things, but.. That would not be too secure. From the cached object, a client could spy on any object anywhere near them. They could also get all of their in game stats (which will not be allowed).

My solution was to use the Manipulators (the things I spoke of below when I couldn’t remember the cool name I had for them). A manipulator is an interface that is to an object that you can use to control said object. You could have a CarManipulator that let you step on the gas, brake, and so forth. In this case, objects that are playable (i.e. a client can connect to it) have a PlayerManipulator type object. That will be used to control the player’s character. In addition, you also get a PlayerView cache (the original is stored on the PlayerServer). The PlayerServer will put objects into the view when– you guessed it– they come into view. It takes them out when they’re gone. It won’t even put real objects in there, but ObjectViews which will be the things that actually know how to interact with Ogre to show up on a client’s screen.

I also spent a bit of time thinking about Terrains and how they’ll work. I’m still not really sure. The old MV3D way of doing it was going to be to send only terrain blocks nearby the player and to send more detail for blocks that were closer. I don’t think that would work. Either way, I have more design to do on that end.

I’ve just finished the unit testing for some Very Cool Stuff ™. I now have a general purpose database loading and saving mechanism. It can save any Python class which subclasses Persistable (a cool thing in its own right) with minimal effort. All you need to do is subclass, set which member variable is your object identifier, and make sure to give your object’s constructor along with any sub objects constructors to the database persister. With that in place, you can load and save your object in the database by calling yourobject.Write() or SQLGeneral.Read(instanceid=(your object’s identifier)).. It’ll handle persisting everything in your object. It can even do multiply nested lists/dicts/tuples and properly handles circular references. (Object a includes object b, and object b includes object a) The only thing left to do with it is to make it work with Twisted’s Async DB stuff. You can even have multiple separate object repositories within the same database.

The whole thing uses 5 tables. Oh did I mention, that if you go and add more properties to an object you already saved, it doesn’t care in the least? As long as it has your constructor and your unique identifier, you are good to go. You can also extend it past the types it understands (string, int, float, array, dict, Persistable object, Date) if you have say objects that you want to save using a different DB schema.

If your head is spinning and you have no idea what I’m talking about, let me try to explain. In an MMORPG, since there are so many objects, places, and other stuff, you can’t really get away with saving everything in files on a hard drive. It would be way too slow. So everyone uses databases. You can think of a database as an Excel spreadsheet. Each individual worksheet is a database table, and instead of having unlimited columns, you have to setup the number of columns (and their names) in advance and more or less stick to that. In Multiverse3D, there will be many types of objects. Each one has different properties. For example, a person may have a hair color property, while a lamp may have a turned-on property which specifies if it is currently on. Most MMORPGs (as far as I’m aware) either railroad all objects into one database table (either by having lots of columns and wasting space, or by some clever way of representing the possible attributes in the columns), or have database tables that only keep properties with other ones to keep the core object data (properties that all objects have like position in the game world). Anyway, if you make any changes or need to add a special property, chances are, they are going to have to do some major DB work (by hand). My code (just under 500 lines) takes a standard set of simple database tables (5 tables with the widest having 10 columns) and lets you use those to save an object with any properties whatsoever. No need to change the tables by hand, and the code does not change the columns in the tables at all.

Before I set out to program this, I did do some research to see if what I wanted existed, and as far as I can tell, it does not. Would have been nice if it did because then I wouldn’t have spent the last week on it.
Next on the list is a similar XML persister. I already have a Pickle persister (using Persistable), but that’s nothing to write home about as it’s maybe 30 lines long. XML persister (which will also handle configuration files.. Then maybe make a Server executable and define the PlayerInterface (and player movement), maybe do a little with object factories, some terrain stuff if I get very ambitious. Once all that is squared away (probably the most complex is the player movement since making players move in a physics world is complicated), then I’ll start work on the client. I was hoping to start this weekend, but that’s probably not going to happen.

I spent some time this morning evaluating Ogre, a 3D library alternative to Crystal Space. Not sure. Some parts of it are a lot more advanced than Crystal Space, but in some ways, it appears somewhat inflexible. It does work in Python and on Windows & Linux, so maybe I’ll have a closer look (i.e. install it) when it is time to work on clients.

Update: 8:14PM. 4 hours to impliment a similar persist mechanism that saves to XML. Not bad– especially considering this is the first time I’ve really worked with XML parsing.

I started writing stress tests and am now sort of glad that I did. I found out that I can drop 1000 balls onto the ground and get about 4 fps on a P3 550 server, but when another server instance (running on the computer) is involved just getting updates on those balls, twisted breaks. I hit some sort of maximum cached objects limit. Suck!

Back down to 100 balls and single server gets a respectable 50fps (desired server FPS is 20, so this is 2.5x the desired performance), but dual servers gives me 1.5fps. Suck! Again!

I will say that from experience, simulating 1000 balls hitting the ground on a P4 2.4Ghz (including 3d Graphics) gets you 3fps I believe, but a good chunk of that is graphics.

So I don’t know what to do about problem A really (I mean, I don’t see what the problem with having 1000’s of cached objects is. Hopefully there is just some setting to increase that number.), problem B will require some checking into of the network code to see if the slowness is with Twisted, or if it is with my code. Hopefully it’s my code. I mean, when I send updates to object positions over the network, I don’t exactly do it in a cpu or network speedy way. Every frame where the objects position, rotation, linear velocity, or angular velocity have changed, they are sent across the wire as 3 floats (for each that has changed). Not too smart, but I didn’t intend it to be just yet.

Well, off to fix A&B.

Update: Fixed A (as noted in comments), B has been tracked down to Twisted’s callRemote function. Something that should return immediately in theory, but aparently doesn’t.

Here’s some output (the most frequent calls to callRemote were commented out):

08/15/2006 10:46:43 - Stress_Physics: Stress_CollidingObjects:
  IPS 1x 138.18 100x 112.33 1000x 15.75 5000x 0.28
08/15/2006 10:46:43 - Stress_Physics: Stress_CollidingObjectsNetwork:
  IPS 1x 60.88 100x 110.99 1000x 8.84 5000x 0.24

IPS=Iterations (frames) per second
1x, 100x, etc= how many objects were dropped

5000 Objects is clearly more than the server can handle. About 800 sounds like the right number. Still, out of 328 seconds of testing, only 22 of those seconds were spent on physics. Which leaves 300+ seconds of Twisted and my code. Since there were 315 iterations, the physics alone averaged 14 iterations per second. (1/(22/315))

“Don’t put that there!” — Prince Alexi (Jay Bardell) in The Cheezy AD&D Movie

So why they quote? Well, I thought it was a good fit since I’m reorganizing lots of stuff in the new MV3D code. Well mostly servers. I made the change I mentioned the other night but didn’t move all the code to the new ServerInterfaces. I’m doing that now. I’m also trying to come up with a good directory structure. While it’s not as important to have the directory structure set in stone now that I’m using SVN instead of CVS, it’s still a pain in the ass to change it because all of the import statements in Python have to be changed.

In addition, I am removing the wierd MasterServer code that looks up realms and stuff and replacing it with a generalized DirectoryServer that stores directories of where objects live.

There are some cool new additions to Cacheable things– things which can be sent over the network once but will be updated by their parent server. Now they automatically keep track of how cached they are. Since I’m designing things so that you can take a Cacheable object and send it from server A to server B then server B can send the object to C. Whenever there’s a change, the change goes from A->B->C. B would have a cached count of 1 while C’s would be 2. I’m also trying to fit in a way for B or C to update A (assuming they have the permission to).

Damn, what was I calling them? I had a great name. Well, until I remember the name, they’re called controllers. Objects can have a bunch of controllers and they are passed along with the object when it is cached. The server that has the cached object can use the controller interface to affect the object. For instance, a car may have a DriveInterface which lets you step on the gas, brake, or turn the steering wheel. See what I’m getting at? Player objects will have interfaces like this as well that the Client will use to move their PC around. Since I can’t remember, if anyone has a good name, let me know.

Tonight I wrote the basis for the Asset management system. Assets are images, sounds, 3d models, or any other media used by the game. I’m fairly pleased though just now I’ve decided I may want to re-work how server interfaces go. More on that later. The asset system seems like it’ll be really cool. Let me just re-iterate how nifty Twisted stuff is– and also how much Python rulz!

So there is an Asset class and that’s the only thing that’ll get transferred directly to the client. The asset class is subclassable, and through that means, you get to define how the asset is acquired. Sure, there will be HTTP, FTP, SCP, and SFTP, but this means the possibility is there for more complicated things like BitTorrent (many clients of which are written in Python by the way). Granted, BitTorrent doesn’t really lend itself to Assets you need very quickly (like images or models), but it could be good for say downloading a whole chunk of Assets at once or a new version of the software (some parts of the software — maybe all — will be under Asset control). You could also have the Asset be an audio or video stream and have a live broadcast right into the game.

Anyway, enough avout that. I think it’ll work pretty well. The server side of it mirrors how Objects and Realms work with Assets being like Objects and AssetGroups being like Realms. When I get to the client side, I’ll work on a user-adjustable cache of assets like old MV3D had (oh did I mention that in the features? That was a pretty nice one. It had a variable sized disk and memory cache. When the memory cache got too big, it put stuff on disk and when the disk cache got too big, it erased older/less used stuff. It even worked most of the time.)

So right now when you request a ServerInterface (say a “Realm Server Interface”), you get a new RealmServerInterface instance created on the server. Those *Interfaces don’t have any data other than a pointer to the server they live on, and all they do when you call methods on them is to query that server. Well, I think it would be nice (and a lot cleaner) to allow a Server to have a persistant set of interfaces that store the data related to themselves (RealmServerInterfaces store Realms for example) and when you request a RealmServerInterface, you get a remote reference (Viewable) to that instance of the interface instead of a new instance. I think that’ll clean up the server code a bit since it is getting messy.

Oh yeah, and I did start catching up on my documenting. Got up to the letter C in the list of source code files.

So I decided to throw out the hundred thousand lines of code in the old MV3D and start over. Sounds silly, I know, but taking the old code and fixing it would be much more of an effort. For this version, I have a pretty strict plan of how I want to go about programming. For each feature, there should be four phases: plan, document, impliment, and test. They should happen in that order. I feel that if I stick to that method, I’ll have some pretty good code and will hopefully reduce the amount of refactoring or redesigning I will have to do. I will say that so far, I’ve been doing poorly on the ‘document’ side of things. I’ll have to remedy that. On the flip side, what I have so far is well planned out, written, and tested code. My other ideas on how to make this effort be more successful are to use 3rd party libraries wherever possible, use Python as much as I can, and to concentrate on getting the hard/unfun parts (which basically comprise the foundation of the game) done before moving on to anything.

Did I mention that the new version of MV3D will be open source?

Now I’ve been working on the new MV3D for a few weeks now. It’s at about 5,000 lines and the part I’ve written is all Python. It currently uses Twisted and PyODE (a Python interface to ODE). There are about 70 unit tests written as well (which are included in the 5,000 lines and are almost 3,000 lines). I’m not writing the Client part of things at all right now. I want to get a very stable server with most of the functionality it needs finished before working on the client. The unit tests have been used to make sure everything is doing what I think it is doing. Here are just some of the cool things the 2000 lines of server code can do: