MV3D Development Blog

No updates for a while probably.. I was trying to compile PyOgre (the Python Ogre module) on Linux and kept running out of RAM. I only had 512mb because my motherboard used rambus memory and it is prohibitively expensive to buy. Instead, I bought a new motherboard and CPU fan for about 1/2 the price of getting another 512mb of Rambus memory. Rambus=teh sux0rz. Anyway, installed the motherboard last night and Windows XP refuses to even attempt to boot (bluescreen before seeing anything else). Linux pretty much works fine, but it has decided that my ATARAID controller card is the devil. Perhaps that’s Windows problem too since Linux does an Oops* or something when trying to mount a drive off the RAID controller.

So… That plus the fact that I can’t upgrade my Linux kernel to v2.6 because of the RAID controller is making me reconsider using it. It’s crap anyway because the hardware is just an IDE controller and the drivers for it do all the RAID in software. So I’m probably better off doing software RAID in linux/windows. Just I think they don’t play nice with each other if you want to use one RAID array that has both Windows and Linux partitions on it (like I’m doing now). I think I’ve come to the conclusion that I don’t need RAID in Linux. I don’t store anything useful on my Linux box really. All the source code lives in the SVN repository on another system (which uses a similar ATARAID card, but doesn’t have Windows on it or any need to be upgraded to 2.6 kernels).

I’m considering rebuilding both my Windows and Linux installs on my computer since they are both screwed up in other ways as well. Since I wanted to keep my previous Windows 2000 install until I was finished upgrading and was sure it would work, my Windows XP install got stuck on what it calls the E: drive. And I have a C: drive that has Win2k on it which I never use and can’t get rid of because WinXP thinks that’ll cause the end of the world. (It probably will because it also is my boot drive with LILO on it, but Windows doesn’t know that)

Hopefully I can use the RAID controller as an IDE card because I do need to keep RAID in Windows, and the onboard ID supports 4 drives (DVDR, Boot Drive, and 2 large drives for my video production biz).. In Linux, I installed VMWare tools when I was running that installation as a VMWare client OS on Windows. Now it screws up my X settings every time I boot (it didn’t do this a few days ago which is odd since I haven’t used VMWare on it in years).

That’s all for now other than almost buying a new shiny car the other day. Stupid car dealers. They suck.

* Oops: Seriously, linux Kernel errors are funny. It really says oops. One of my favorites (which I have not personally seen) is “LP1 on fire!” (as in “the printer, the printer, the printer’s on fire! We don’t need no water….”)

Ogre may not be such a sure thing. I took the first Ogre tutorial and put in some simple MV3D code that uses Twisted to connect to a MV3D server. Unfortunately, it seems like Ogre (or PyOgre) messes with the system timer causing time to basically stand still for the Ogre application. Twisted was very confused by this and kept saying the connection timed out immediately. Using Twisted to schedule a function to be run in say 5 seconds caused that function to run immediately. So it pretty much makes Twisted useless. This is no fault of Twisted’s because if I use Python’s time and date modules while running an Ogre app, they return the same time. I posted to the PyOgre forums about it, so we’ll see what happens. If no one has a clue or it’s a bug that can’t be easily fixed, then Crystal Space it is. I also want to see if the problem happens in Linux since I suspect it doesn’t. Other people are using PyOgre+Twisted, so I have to assume either it works in Linux, or something is borked with my Windows setup other than Windows being borked in the first place.

Speaking of borked though, that also describes my Linux partition on my desktop. Well, it was mostly working last night, but Debian was telling me that I basically had to do a dist-upgrade in order to install any new packages, so I’m doing a dist-upgrade now.

Aside from the time problem, Ogre does not have a suitable Terrain system nor does it have portals at all. :-/ Actually, I think the major difference between it and Crystal Space is that Ogre has a lot less features, but they are better done. Crystal Space has portals of course and a few Terrain types to choose from. I want the good parts of them both, dammit! Ogre also has pretty good documentation. With Crystal Space, it’s “Isn’t the code documentation enough? I mean, there’s almost half a million lines of it for you to read through!”

Well, maybe this is someone trying to tell me to finish up more Server related code like auto-perist to database and load balancing of already loaded areas (load balancing of new areas is in place already).

I did some more playing with Ogre today including downloading the demos. They were about 35MB, which seemed odd to me since the bare essentials of Crystal Space is 200MB (which compresses to about 50MB in a zip file). The demos were very good and showed off quite a few features that Crystal Space just doesn’t have. I went on to download the Python development environment for Ogre. Once again, I was confused as these were only 5MB. I figured I must have to download the rest of Ogre to use them (sort of like Crystal Space’s Python implimentation). Nope. 5MB and that’s it (plus 12mb for some demo data like models and such that was a seperate download). I installed it (and the prereqs– python 2.4, and psyco) and after having to download a random DirectX DLL, I was able to run the demo apps. Last time I got Crystal Space working on Windows, it took me about a week. This was an hour tops. So I think pending an investigation on whether Ogre supports some required things like Portals (which it claims to), I’ll probably be using it for MV3D.

That said, clearly, I’ve been working on client stuff today. Just for kicks, I tried running a server on Windows and after some major tweaks to my Date class, it works. Dates are still way screwed up, but I can run the unit tests (except MySQL related ones since I didn’t install the Python MySQL module– the client won’t use that).

I figured out what sort of view clients are going to have on things finally as well. I’d been thinking that I would just give them a cache of the object they control and then have some sort of interface for them to send commands and request things, but.. That would not be too secure. From the cached object, a client could spy on any object anywhere near them. They could also get all of their in game stats (which will not be allowed).

My solution was to use the Manipulators (the things I spoke of below when I couldn’t remember the cool name I had for them). A manipulator is an interface that is to an object that you can use to control said object. You could have a CarManipulator that let you step on the gas, brake, and so forth. In this case, objects that are playable (i.e. a client can connect to it) have a PlayerManipulator type object. That will be used to control the player’s character. In addition, you also get a PlayerView cache (the original is stored on the PlayerServer). The PlayerServer will put objects into the view when– you guessed it– they come into view. It takes them out when they’re gone. It won’t even put real objects in there, but ObjectViews which will be the things that actually know how to interact with Ogre to show up on a client’s screen.

I also spent a bit of time thinking about Terrains and how they’ll work. I’m still not really sure. The old MV3D way of doing it was going to be to send only terrain blocks nearby the player and to send more detail for blocks that were closer. I don’t think that would work. Either way, I have more design to do on that end.

I’ve just finished the unit testing for some Very Cool Stuff ™. I now have a general purpose database loading and saving mechanism. It can save any Python class which subclasses Persistable (a cool thing in its own right) with minimal effort. All you need to do is subclass, set which member variable is your object identifier, and make sure to give your object’s constructor along with any sub objects constructors to the database persister. With that in place, you can load and save your object in the database by calling yourobject.Write() or SQLGeneral.Read(instanceid=(your object’s identifier)).. It’ll handle persisting everything in your object. It can even do multiply nested lists/dicts/tuples and properly handles circular references. (Object a includes object b, and object b includes object a) The only thing left to do with it is to make it work with Twisted’s Async DB stuff. You can even have multiple separate object repositories within the same database.

The whole thing uses 5 tables. Oh did I mention, that if you go and add more properties to an object you already saved, it doesn’t care in the least? As long as it has your constructor and your unique identifier, you are good to go. You can also extend it past the types it understands (string, int, float, array, dict, Persistable object, Date) if you have say objects that you want to save using a different DB schema.

If your head is spinning and you have no idea what I’m talking about, let me try to explain. In an MMORPG, since there are so many objects, places, and other stuff, you can’t really get away with saving everything in files on a hard drive. It would be way too slow. So everyone uses databases. You can think of a database as an Excel spreadsheet. Each individual worksheet is a database table, and instead of having unlimited columns, you have to setup the number of columns (and their names) in advance and more or less stick to that. In Multiverse3D, there will be many types of objects. Each one has different properties. For example, a person may have a hair color property, while a lamp may have a turned-on property which specifies if it is currently on. Most MMORPGs (as far as I’m aware) either railroad all objects into one database table (either by having lots of columns and wasting space, or by some clever way of representing the possible attributes in the columns), or have database tables that only keep properties with other ones to keep the core object data (properties that all objects have like position in the game world). Anyway, if you make any changes or need to add a special property, chances are, they are going to have to do some major DB work (by hand). My code (just under 500 lines) takes a standard set of simple database tables (5 tables with the widest having 10 columns) and lets you use those to save an object with any properties whatsoever. No need to change the tables by hand, and the code does not change the columns in the tables at all.

Before I set out to program this, I did do some research to see if what I wanted existed, and as far as I can tell, it does not. Would have been nice if it did because then I wouldn’t have spent the last week on it.
Next on the list is a similar XML persister. I already have a Pickle persister (using Persistable), but that’s nothing to write home about as it’s maybe 30 lines long. XML persister (which will also handle configuration files.. Then maybe make a Server executable and define the PlayerInterface (and player movement), maybe do a little with object factories, some terrain stuff if I get very ambitious. Once all that is squared away (probably the most complex is the player movement since making players move in a physics world is complicated), then I’ll start work on the client. I was hoping to start this weekend, but that’s probably not going to happen.

I spent some time this morning evaluating Ogre, a 3D library alternative to Crystal Space. Not sure. Some parts of it are a lot more advanced than Crystal Space, but in some ways, it appears somewhat inflexible. It does work in Python and on Windows & Linux, so maybe I’ll have a closer look (i.e. install it) when it is time to work on clients.

Update: 8:14PM. 4 hours to impliment a similar persist mechanism that saves to XML. Not bad– especially considering this is the first time I’ve really worked with XML parsing.

I started writing stress tests and am now sort of glad that I did. I found out that I can drop 1000 balls onto the ground and get about 4 fps on a P3 550 server, but when another server instance (running on the computer) is involved just getting updates on those balls, twisted breaks. I hit some sort of maximum cached objects limit. Suck!

Back down to 100 balls and single server gets a respectable 50fps (desired server FPS is 20, so this is 2.5x the desired performance), but dual servers gives me 1.5fps. Suck! Again!

I will say that from experience, simulating 1000 balls hitting the ground on a P4 2.4Ghz (including 3d Graphics) gets you 3fps I believe, but a good chunk of that is graphics.

So I don’t know what to do about problem A really (I mean, I don’t see what the problem with having 1000’s of cached objects is. Hopefully there is just some setting to increase that number.), problem B will require some checking into of the network code to see if the slowness is with Twisted, or if it is with my code. Hopefully it’s my code. I mean, when I send updates to object positions over the network, I don’t exactly do it in a cpu or network speedy way. Every frame where the objects position, rotation, linear velocity, or angular velocity have changed, they are sent across the wire as 3 floats (for each that has changed). Not too smart, but I didn’t intend it to be just yet.

Well, off to fix A&B.

Update: Fixed A (as noted in comments), B has been tracked down to Twisted’s callRemote function. Something that should return immediately in theory, but aparently doesn’t.

Here’s some output (the most frequent calls to callRemote were commented out):

08/15/2006 10:46:43 - Stress_Physics: Stress_CollidingObjects:
  IPS 1x 138.18 100x 112.33 1000x 15.75 5000x 0.28
08/15/2006 10:46:43 - Stress_Physics: Stress_CollidingObjectsNetwork:
  IPS 1x 60.88 100x 110.99 1000x 8.84 5000x 0.24

IPS=Iterations (frames) per second
1x, 100x, etc= how many objects were dropped

5000 Objects is clearly more than the server can handle. About 800 sounds like the right number. Still, out of 328 seconds of testing, only 22 of those seconds were spent on physics. Which leaves 300+ seconds of Twisted and my code. Since there were 315 iterations, the physics alone averaged 14 iterations per second. (1/(22/315))

“Don’t put that there!” — Prince Alexi (Jay Bardell) in The Cheezy AD&D Movie

So why they quote? Well, I thought it was a good fit since I’m reorganizing lots of stuff in the new MV3D code. Well mostly servers. I made the change I mentioned the other night but didn’t move all the code to the new ServerInterfaces. I’m doing that now. I’m also trying to come up with a good directory structure. While it’s not as important to have the directory structure set in stone now that I’m using SVN instead of CVS, it’s still a pain in the ass to change it because all of the import statements in Python have to be changed.

In addition, I am removing the wierd MasterServer code that looks up realms and stuff and replacing it with a generalized DirectoryServer that stores directories of where objects live.

There are some cool new additions to Cacheable things– things which can be sent over the network once but will be updated by their parent server. Now they automatically keep track of how cached they are. Since I’m designing things so that you can take a Cacheable object and send it from server A to server B then server B can send the object to C. Whenever there’s a change, the change goes from A->B->C. B would have a cached count of 1 while C’s would be 2. I’m also trying to fit in a way for B or C to update A (assuming they have the permission to).

Damn, what was I calling them? I had a great name. Well, until I remember the name, they’re called controllers. Objects can have a bunch of controllers and they are passed along with the object when it is cached. The server that has the cached object can use the controller interface to affect the object. For instance, a car may have a DriveInterface which lets you step on the gas, brake, or turn the steering wheel. See what I’m getting at? Player objects will have interfaces like this as well that the Client will use to move their PC around. Since I can’t remember, if anyone has a good name, let me know.

Tonight I wrote the basis for the Asset management system. Assets are images, sounds, 3d models, or any other media used by the game. I’m fairly pleased though just now I’ve decided I may want to re-work how server interfaces go. More on that later. The asset system seems like it’ll be really cool. Let me just re-iterate how nifty Twisted stuff is– and also how much Python rulz!

So there is an Asset class and that’s the only thing that’ll get transferred directly to the client. The asset class is subclassable, and through that means, you get to define how the asset is acquired. Sure, there will be HTTP, FTP, SCP, and SFTP, but this means the possibility is there for more complicated things like BitTorrent (many clients of which are written in Python by the way). Granted, BitTorrent doesn’t really lend itself to Assets you need very quickly (like images or models), but it could be good for say downloading a whole chunk of Assets at once or a new version of the software (some parts of the software — maybe all — will be under Asset control). You could also have the Asset be an audio or video stream and have a live broadcast right into the game.

Anyway, enough avout that. I think it’ll work pretty well. The server side of it mirrors how Objects and Realms work with Assets being like Objects and AssetGroups being like Realms. When I get to the client side, I’ll work on a user-adjustable cache of assets like old MV3D had (oh did I mention that in the features? That was a pretty nice one. It had a variable sized disk and memory cache. When the memory cache got too big, it put stuff on disk and when the disk cache got too big, it erased older/less used stuff. It even worked most of the time.)

So right now when you request a ServerInterface (say a “Realm Server Interface”), you get a new RealmServerInterface instance created on the server. Those *Interfaces don’t have any data other than a pointer to the server they live on, and all they do when you call methods on them is to query that server. Well, I think it would be nice (and a lot cleaner) to allow a Server to have a persistant set of interfaces that store the data related to themselves (RealmServerInterfaces store Realms for example) and when you request a RealmServerInterface, you get a remote reference (Viewable) to that instance of the interface instead of a new instance. I think that’ll clean up the server code a bit since it is getting messy.

Oh yeah, and I did start catching up on my documenting. Got up to the letter C in the list of source code files.

So I decided to throw out the hundred thousand lines of code in the old MV3D and start over. Sounds silly, I know, but taking the old code and fixing it would be much more of an effort. For this version, I have a pretty strict plan of how I want to go about programming. For each feature, there should be four phases: plan, document, impliment, and test. They should happen in that order. I feel that if I stick to that method, I’ll have some pretty good code and will hopefully reduce the amount of refactoring or redesigning I will have to do. I will say that so far, I’ve been doing poorly on the ‘document’ side of things. I’ll have to remedy that. On the flip side, what I have so far is well planned out, written, and tested code. My other ideas on how to make this effort be more successful are to use 3rd party libraries wherever possible, use Python as much as I can, and to concentrate on getting the hard/unfun parts (which basically comprise the foundation of the game) done before moving on to anything.

Did I mention that the new version of MV3D will be open source?

Now I’ve been working on the new MV3D for a few weeks now. It’s at about 5,000 lines and the part I’ve written is all Python. It currently uses Twisted and PyODE (a Python interface to ODE). There are about 70 unit tests written as well (which are included in the 5,000 lines and are almost 3,000 lines). I’m not writing the Client part of things at all right now. I want to get a very stable server with most of the functionality it needs finished before working on the client. The unit tests have been used to make sure everything is doing what I think it is doing. Here are just some of the cool things the 2000 lines of server code can do:

So clearly, I don’t post very often. Not much to write about really. However, I think I may have found a solution to that problem. Unfortunately for anyone reading, my solution is to use this as a place to write about what’s going on with the computer game I’m writing– Multiverse3D.

For anyone who doesn’t know, I should give a brief history. MV3D is a 3D MMORPG (Massively Multiplayer Online Roleplaying Game) that I’ve been writing on and off since 1997 or so. It was originally based on a text only MUD (multi user dungeon) game that I wrote/ran in the mid 90’s. The concept was to have a game engine that was capable of simulating worlds of all types and not something that was just stuck in one genre (i.e fantasy/sci-fi/etc). The worlds could be linked together so that you could have fun interactions between them. At the height, I managed a volunteer team of about 10 people– artists, programmers, designers, etc.

There were a lot of issues with the previous iteration of the game. One of the bigger ones was that you can’t really rely on free help. Granted, there were some people who I could most certainly rely on, the majority of people who joined the project ended up sucking up a lot of my time in training them before they went inactive or left. Most of the technical problems involved times when as the ‘head honcho’, I looked a daunting problem in the eye and said ‘let’s figure that out later’. Other ones were just from lack of experience. Creating a MMORPG is a lot different than creating a regular computer game. On the whole, I would have to say there was no way that the 110,000+ lines of code that make up that effort could go anywhere useful.

The state of affairs last time I worked on that code was that you could run a server, create a basic scene (say, a small landscape and a few objects), connect to it with a single client, walk around (though you would pass through walls), and that’s it. Even with just one client connected, it was horrendously slow. Lag times with the server running on the local machine were close to one second. Now that sounds like very little for 110K lines of code to be doing– especially if you consider that none of the physics (ODE) or graphics (Crystal Space) were handled by the 110KLoC. There was a lot of stuff going on in the background such as instance management, asset management (including autodownloading content during gameplay), and so on. The client and server would crash very frequently– a milestone goal was set to get the server to run for more than a day without crashing.

It wasn’t all bad though. There were some good parts, and I (and hopefully the rest of the team) all learned a lot in the process. Here’s a short list of what I think are the good aspects:

Bad concepts of accounts. Player accounts were never integrated properly. Another important feature. An account should be what has the users password and list of PCs as well as their billing info. (maybe) Not so much.