MV3D Development Blog

I made a small amount of progress last night. For the first time, I was actually able to start a server that uses the new OctreeArea and connect to it with a client. I could even walk around. Unfortunately, collision detection seemed like it was somewhat messed up. On the other hand it seems like objects can move from one octree node to another– but maybe that was just an odd bug which acted like a feature. Node splitting is definitely broken. It split the parent node twice so that it ended up with 16 branches where it was only supposed to have 8.

I was playing with 3DS Max for the videos I was working on and decided to try exporting stuff to Ogre. It was ok. Static models were a breeze to export. No problems whatsoever. There is even a physics exporter as part of the Octopus plugin. It could be just what I need for physics properties. Anyway, it wasn’t all good unfortunately. I wasn’t able to export any models that had animation such as player models. The exporter bugs out for some reason while saving the skeletons or something. This will have to be figured out relatively soon.

The videos are done, so I’m now getting back into MV3D stuff.

And the idiot of the day award for yesterday goes to the electrician who was overheard saying “Hey! Maybe this big red button in the server room is the air conditioner’s power switch I’m looking for!” Just before lifting the protective plastic cover around the button and pushing it. That was the Emergency Power Off (EPO) button for our server room which causes the UPS to physically disconnect its batteries and switch itself off. Needless to say, this is not very good for servers. I didn’t get everything back online until late last night. What’s even better is that the electrician’s company installed the EPO button in the first place about 4 months ago.

Is it just me, or is a big red button under a plastic cover not something that should just be pressed for the heck of it? Even if you know nothing about electrical codes (and why there needs to be a big red button in a server room and all), big red buttons are not something to hit unless it’s an emergency and you know what you’re doing.

BTW, if you are wondering, the red button is there in case of a fire. It is pressed to make sure that the combination of water from the sprinklers and power from the UPS will not electrocute any firemen. Running servers while they are getting wet is also a bad idea. They have a much better chance of surviving a fire if they are shut off before they get wet– assuming it’s not the servers that are on fire.

Well, I just realized both MV3D servers (including the one that hosts the development website) aren’t totally back up after a power outage the other day (got to install that UPS!), so instead of posting this to the dev website, I’m forced to write it here. Oh the horror! Sorry if I jump around or write about steps a and f, but miss b,c,d, and e. You should be used to that if you read stuff here often though.
I just did a little research, and it seems like it would be no problem for me to create a view setup similar to any 3D modeling program within Ogre. I can split the screen into 4 boxes which can show views from the top, side, front, and what your character sees. The top/side/front views could easily be set to wireframe as well so that you can see through objects. This will probably be useful in World Editing, though I think I may make it an easy option to switch into (say hit F10 or something to swap between normal view and that). Placing and moving objects with those views should be a snap because it would be very easy to line them up. The cameras for all the views should be independently controllable just like a 3d program.

Next, ideas for creating objects. I need to create an asset type for ClassGenerators if there isn’t one because when creating an object in MV3D, you just need an area to put it in, and a ClassGenerator. The ClassGenerator specifies the Python class that is becomes the object. More on assets in a bit. Somehow, you’d need to be able to select an asset to use as the object. That would require an asset selector. The asset selector should probably show assets from a particular asset group and should be configurable so that only assets of certain types are selectable. So that way, if you specified it should only show object creating assets, it would do that. Anyway, once you select the asset, it should let you place it.

Editing objects. All Objects should define a GUI for editing their properties. Since the properties for each type of object are completely different, this is definitely necessary. Part of me wonders if it would be best to generate the GUI in code, or even dynamically. Doing it without that would mean that developers would have to design (by hand) the GUI and code it for each object type, making creating new object types more of a pain. It would be easier on the developer to inherit from somewhere and just add to the properties GUI from there; however, that may cause retarded looking GUIs. Either way, a custom GUI for each object is necessary because for instance, a PC type object would have editable properties for health, experience, and so forth while a rock sitting on the ground would not.

Defining assets. This could be something that is best left outside the game. Many things in MV3D are specified by an asset (which is part of an asset group). For instance, images, models, sounds, pieces of code, and GUI layouts would all be examples of assets. Asset groups are just a way for people to say “these assets are part of Mike’s medieval fantasy pack” or whatnot. Anyway, the assets themselves will always be edited/created/etc outside of the game in 3D software, text editors, GUI editors, etc. It probably makes sense to have some sort of program that lets asset creators add, modify, and delete assets in a simple fashion that connects to a game server in order to register the assets within the game automatically….. oh… DUH! This sounds like the job for a WebApp. Asset creators could log in to it, upload the asset to the web app, and edit all the asset metadata. Cool. A good first step would probably to just be able to edit asset metadata. One issue is that there can be any number of asset types which can have completely different metadata. For a silly example, one asset type downloads via HTTP/FTP so it requires a URL. Another one may use some peer2peer mechanism to download and require whatever info for that. So, there needs to be a way to generate and parse HTML forms based on the structure of an asset object. I think that’s do-able. Ok, way off subject here. But first, you know what would also be cool? If you could dump a .3ds file on the web app and it would run Ogre’s conversion utility to generate the Ogre .mesh file and stuff. That would be cool.

The other major component of the world editor will be related to modifying the terrain. I’ve discussed this a bit in other places, but the general idea for terrain height modification is to show a greyscale image where lighter colors represent higher points on the terrain and allow the user to ‘paint’ over it with various operations. I think the image shown should not be directly used in the heightmap as each pixel in the image is limited in range between 0 and 255 and is an integer whereas the value for the height of that image can be a decimal number between… the extents of a 32bit float. (-a lot to +a lot) I think in the old MV3D, I made 0 be the lowest point in the terrain and 255 be the highest and everything in between would scale accordingly. That’s probably a good idea still. It would probably also be nice to be able to adjust the points in the 3d view as well.

In addition to adjusting the heights of the terrain, people will need to paint things on it such as roads, grass, sand, rocks, etc. (I’m talking flat texture maps on the terrain) For this, I’ll need the splat material compositor I’ve been thinking about working, but other than that, it could work similarly to the height editor with its image that you draw on. Painting directly to the 3d view should work as well. In addition to flat things, there will be non flat things such as vegetation. Once again, this could work similarly to the height editor.

One problem with editing the terrain will be that MV3D’s current terrain consists of a grid of terrain blocks which are not really connected to each other. Somehow, the editor will have allow you to modify the heights of one terrain along the edge, the heights on the corresponding edge of the terrain next to it will need to be modified as well.

I just had an interesting thought. Maybe it makes sense to not separate out terrain from regular objects. In MV3D, terrain is a regular object. Some places won’t have terrain even. So, all these terrain editing details could be part of the terrain object’s properties. That actually could work very well because some terrains will have features for grass and such while others may not.

Finally, not all objects in an area are viewable on the client. Some things just don’t have a visual representation. That makes them a little hard to select with the mouse. It should be possible to pop up a window that lists the content of the area and lets you select things. Once select something, it jumps into the properties of that object. When editing those objects, it may not even be possible to place them, so placing objects will have to be dependent on the type of object. That means the placement code should be part of the object properties GUI.

Ok, that’s probably enough blabbing for now. This is going to be a lot of work to put together, but it should be pretty kick-ass in the end.

The lack of updates lately is due to there being not much progress. I believe I mentioned earlier that I have two video jobs I’m working on this month. Pretty much all of my non work time has been devoted to that.

I did do a couple of things such as make a small change to the DB Persisting routines that makes them about 10x faster. Still, that’s not saying a heck of a lot. They are slightly more usable now though.

The current plan is to work more on the Octree based Area code until it is at the point where objects can be placed in it and can move around. They’ll need to move from node to node as appropriate as well. Once that’s done, I’ll take a break from it and work on some of the world editing features. I’d really like to get those up to speed so that I can start building simple worlds. Hopefully, I’ll be able to keep the worlds persisted and use them for various testing efforts. Either way, this may require speeding up the DB Persisting stuff even further. Another 10x speed increase would probably be good. I’m just not sure how I can get that at this point as it seems I’ve sacrificed speed for flexibility in a big way here. I can persist any Python object to DB, but it takes for frikkin’ ever. Currently, it uses 4 tables: instances, properties, arrays, and dictionaries. I could cut out properties really because properties basically stores object.__dict__, which I’m sure even the non Python minded person could guess is a dictionary. Therefore, properties could be saved in dictionaries, but I don’t think that’ll speed anything up (and it could be a fair amount of work). I should probably not be lazy and do it because if it gets done now, that means no one will have to convert database data from the old way to the new.

The only other idea that I have for the database is to possibly start doing some batch transactions. That may speed things up a bit. This could also be pretty tricky because while you’d think you could just batch up all the transactions to write an array to disk, you can’t do that if the array contains an instance or something else that requires getting the next ID number from the database. You could keep track of all the ID numbers, but that limits you to only having one thread writing to the DB at a time. Maybe that solution is ok. I’ll have to think about it more.

I am, however, getting excited about world editing tools. I was reminded recently about Second Life’s in game editing tools and am thinking that I should really check them out in order to get some ideas. Clearly, their tools work well. Some of the content in there seems to look really nice. Yes, I know Sturgeons Revelation and all. But still, it seems like something that deserves looking at. Granted, I do not plan to use the world editor to generate models like in Second Life, but it should mostly be for placing models that have been made elsewhere (blender, 3dsmax, etc).

Part of the problem is that it’s hard to align things in a 3D view. The other part of the problem is that so far, I’ve yet to see a world editor type app that I really liked. I know what I don’t like, and what’s annoying, but really, there needs to be an intuitive way to do this.

This was too funny to pass up. I was doing a Google search for Multiverse3D, and it came up with this (do not hold me responsible if you click the link):

Damn, why am I wasting my time writing MV3D if there is already a crack and a warez download for it? I could just download that and I’d be finished.

Guess I should be flattered?

Here’s some shiny new screenshots of lights, shadows, water, and me getting totally bored with the 3d models I’ve been using up until now.

An amusing tidbit about the ones with the houses in them: those houses were made from the same object as the white cubes in previous images. I just changed the visual representation. So, in order to get them into the right place, I just kicked them around. It is very amusing to kick a house and watch it go flying (or rolling as the case was). I eventually realized that I had a world editor available to set the position manually, but kicking them around was entertaining.

There are some visual problems with the lighting unfortunately. I think it has to do with the vertex normals on the terrain. Don’t know what the deal is, and I’ve tried pointing them both out and in, so no idea.

Yaaaaay!

Anyway. v0.2 is done, v0.3 is in progress. (yaaay) Man, I’ve got to get some screen shots because character animation, lights, shadows, and water came together really quick. Now water isn’t finished because it isn’t integrated into the physics as I’d like it to be. Even with the Example Ogre water shaders, it looks cool. Oh yeah, so water physics. I think I’m going to try something, but I don’t know if it’ll work. My idea is for bigger bodies of water to define their shape in the physics engine, but instead of having things bounce off of them, a) let whatever comes in contact with it know that it has touched water (so if say you drop a sword into water, it could rust), but more importantly, b) slow the object down and do some very cheezy fake bouyancy tricks. So if you put something in water that is less dense than water, it floats, but otherwise, it sinks. Very fake and simple, but something. Could be cool, could suck, could be too CPU intensive. Should be easy to program.

What I’m working on now are the two more complex problems in v0.3. The one that I would say is mostly done is allowing each area in the game to have its own coordinate system while keeping a seamless transition from one area to the next both physically and visually. It does work at the moment. There is one physical glitch where if you stand right between the two areas, you seem to fall through the ground for some reason, and one visual glitch where there are a couple of frames of discombobulation when passing through the areas. The former should be solveable once I figure out why it’s happening– I am pretty sure I took that problem into account when designing how this all works. The latter may be a little tougher. It’s sort of an order issue. Visual representations are now grouped together in Ogre by area (before everything was all together on the client side), and whole areas can be slid around. Ok, so I didn’t want to get into this, but I think I have to in order to explain fully. Areas are seperated by Gateways, and Gateways can now define a translation and a rotation that happens to objects as they pass through (or see through) the gateway. The translation and rotation specify how to change a given point from Area a’s coordinate system into area b’s. So the server looks through the player’s eyes and finds all the areas they can see and determines that area a is where the player is, so that has a translation of (0,0,0), area b is through a gateway with a (100,0,0) translation, so it’s at (100,0,0). Area c is through a gateway in area b with a (100,0,0) translation, so it is at (200,0,0). It sends all this to the client, and the client (having grouped all of the objects for a given area together) moves them all into their respective offsets.

The problem comes in when you move from area a to b, the server will add (100,0,0) to the PC’s position and send notice of the area change to the PC. When the client gets the position change, it updates accordingly. When it gets the area change, it has to go through all the areas it knows about and adjust their offsets by (-100,0,0). It could just wait for the server to send out offsets again, but the server only does that every few seconds. Anyway, there’s a few frames between changing the position and changing the offsets where either your body is missing, or your surroundings are missing because one or the other hasn’t been updated. Fun stuff. I’ll figure it out. For the moment, I’m living with it and moving on to the other complicated v0.3 thing:

Creating a type of Area that can automatically split itself up amongst several servers. And you thought I hashed load balancing to death in the last update? Oh no! There’s more. Currently, the lowest level that can be load balanced is a whole Area. Under most circumstances, this is perfectly fine. But something that tends to happen in MMOGs is that certain areas/zones become very popular and you get many people hanging around in them. Often, this causes servers or clients to bog down. For now, I don’t care about the client part, but the server part is important. What do you do? Well, you make an area that can distribute pieces of itself across servers. So I am working on an octree type thing that’ll do just this. Somehow, I’ve managed to avoid octrees and quadtrees in my previous programming experience, but they seem pretty neat and are pretty much the way to go for what I’m trying to do. [an octree is a tree structure where each branch has either 8 sub branches or some number of leaves (not both)] The hard part comes in with distributing things. I suspect it’ll be a matter of finding a quick and painless way to change the master server for groups of objects at a time. Currently, if you make a master server into a slave server for an object, it dumps the object and reloads it over the network from the new master server. Not too fast. Hopefully, I can keep the object and somehow reconnect it to the new server.

Finally, I had a very good lunch with and his coworker today. Funnily enough, we compared stories of how slow / annoying our individual methods of persisting random Python objects to database without requiring a specific schema were. We chatted about a lot of other things of course, some of which I may bring up here in the future if they come to fruition. But in any case, it was good to talk to other programmer types about MMO related things.

And finally finally, MV3D dev may slow down a bit over the next month because I’ll be working on two video jobs.

Some cool things have happened in the last few days. The big one is that I was able to successfully have an avatar travel from one area (hosted by server A) into another area (hosted by server B). It is completely seamless as well. That is definitely a big step, and it took quite a bit of bugfixing to get working. Basically, it means that the game world can now extend further than the area a single server can host. I was also able to connect to a server with 1000 other avatars standing around. Here is a pretty cool screenshot:

I had to jump up a bit to get a good view. That’s not all 1000; there were several hundred on the other side of the hill as well. The ones are bunched up in the lower left because (I think) they fell down the hill over there. I really have to get rid of the robots though. I will do so after I get them to walk properly (display a walking animation as they move). That’s one of the first things I’ll probably hit on in the next version.

Speaking of the next version. I still have one open bug for this version, but I can’t reproduce it consistently. Sometimes Python prints out an error when you exit the client. That’s all. It’s pretty low priority as it doesn’t really cause any actual problems. I tried a fix, but since I can’t reproduce it on command, I think if it doesn’t happen in the next couple of days, I’ll say v0.2 is done and move on to v0.3.

Here’s another screen shot with 500 avatars and 1000 blocks:

Things seem pretty stable now. As part of the Test Plan, I’ve tested more of the redundancy stuff with a client connected. Now, you don’t really notice losing a server (except the one the client is directly connected to, but that’s a v0.3 feature ). I even tested whacking a login server without issues– it just finds another (automatically even).

I feel like I’ve fixed a million bugs in the last few days.. There were certainly a lot of them. But everything seems pretty good now. I am definitely looking forward to v0.3, though v0.4 is going to be the real fun one. That’s when it starts to become a game.

One more thing. The features that are in place now are almost all of the features a simulation server needs to have. There are a couple more scheduled for the next version or two, most notably: allowing areas to have their own coordinate systems, and breaking a congested area up into pieces to allow multiple servers to split processing of the area between them. Everything else is not strictly simulation based, so it can run on seperate dedicated servers if needs be. This seems to say that Python will work for the server, and that in general, MV3D is very possibly going to work. The foundation of the game is coded and turned out to be very stable. Now that’s a milestone!

Or as B&B put it, “Testies, testies, 1.. 2… 3?”

I’ve got a couple of client related bugs to fix, but then I’ll probably send out the second test version. Maybe a few days, though one bug is definitely a wierd one. All of a sudden, the client started crashing whenever it was starting to display the game world. Other than UI related things and stack thrashing, crashes have been incredibly rare since I’m using Python. I traced this one to the import statement that loads the code to handle the client side of the player’s avatar. Putting a statement before it that prints a test message to the window and does nothing more fixes the problem. I’ve seen this sort of thing many times before in multi-threaded applications. The problem is that the MV3D client isn’t multi-threaded. It makes no sense for a problem like this to occur in a single-threaded app, but well, got to figure that one out.

I was able to increase the server performance by a factor of 10, though. Because of the floating point precision issues I mentioned below, ODE physics does not like big things. I knew when I started using it that the robot model was frikkin’ huge, and I thought maybe it may mess up ODE, but I didn’t want to screw around with scaling in Ogre. Anyway, scaling things by 1/10 allowed me to increase the amount of time that the physics engine could move the simulation ahead each iteration and therefore, went from having to run physics 10 times per iteration to having to do it 1 time. Since collision detection in the physics engine was the reason things were going slow, this bumped the speed up by at least 10 times. In fact, the following screen shot shows the client connected to a world with 2410 objects in it:

I haven’t scaled the landscape down to 1/10 scale yet so it’s blocky, but that’s next on the agenda. I should actually go even smaller because I like using 1 meter = 1 game unit. The robot is 8 game units high. He should probably be more like 2-3. Really, I should ditch the Robot. Soon, oh yes. Soon.

I did play with freeze a bit yesterday. I was able to freeze the client app. Freeze crunches up a Python program in to a Unix executable ELF file. I still have to play around because freeze doesn’t include any pyd/so files.. It also doesn’t know about any things that are imported based on what the server tells it to import. I suspect I can do what I do on the Windows version, which is include the source trees for Client and Util so that the executable can just import out of there. No clue if that works for freeze, but I hope so. It may be hard for me to test as well since I don’t have a second Linux box that has a 3d graphics card in it. I may have to blindly ship it out to a tester unfortunately because any results that I get on my desktop could be invalid. I have all the libraries and python stuff already installed, so the app may work fine on my machine, but fail miserably elsewhere.

I’ve also been thinking about ideas for the next version. In particular, terrain features and in game editing. For the terrain, I have to find a good way to allow for things like roads, paths, and rocky vs grassy surfaces. Right now, each large terrain chunk must have a single texture. What would be cool would be to have a base texture and then be able to overlay other textures on top of it in such a way that they blended at the edges. Then of course, on the terrain comes grass/flowers/small bushes/etc. There is already a sample PyOgre program that makes grass, so I’ll use that.. Or just make a generic way to put stuff on the terrain.

The In Game Editor, which I am creatively calling IGE, is going to be a fun one. It has to be very easy to use, very powerful, and very flexible. A tough combo, certainly. Good UIs are not my strongpoint (even though that was the brunt of my first professional programming job). This is going to require a lot of thinking. If anyone who reads this has any suggestions for how they’d like an in game MORPG world editor to work, please comment.
Finally, if you haven’t seen it already, MMORPG fans or gamers in general will get a kick out of this South Park episode from a few weeks ago. Make Love, not Warcraft. So funny.

V0.2 officially enters the testing phase as soon as I verify that one bug is fixed and fix a second. Yes, I got all of the rest of the features done except for one hat I decided needed more thought, so I postponed it until the next version. The problem mainly was that in order to figure out how the feature should work, things needed to be a little further along then they are. Anyway, I’m excited about the new version. On the surface, it doesn’t really seem like a lot has been added. The client now has terrain and a sky, but nothing much else. Under the hood, though, a lot of things are there that hadn’t been including all the systems for connecting servers together to distribute the game.

Speaking of that, in my last post, I mentioned that I wish I knew how the other MMORPGs did it. Since then, I was able to at least partially answer that question. This white-paper is a very long read, but anyone interested in MMORPG development should consider it. It is a document put together by some of the bigger guys in the MMORPG field which goes over many aspects of creating MMORPGs. It was fairly informative, though a lot of the stuff I already knew or had guessed. It sounds like my method of having lots of little servers simulate the game world is the most flexible way of doing things (although not the easiest). The easiest of course is to throw some big iron at the problem and get a 64/128 CPU server to handle your whole game. That’s a little out of my budget, and when that big honking machine crashes, down goes your game. In my case, it would take quite a few servers going down for clients to see downtime. If one server goes down, the load balancing automatically replaces it. How cool is that? And I’m not talking pie in the sky features that aren’t coded, I’m talking currently finished and tested features. Well, they have to go through the release testing, but they’ve already been internally tested by me. The load balancing works at pretty much all levels. The very top level servers, though, are redundant and have fail over, but can not appoint new top level servers. Fortunately, they work off of a database (or replicated databases), so there can be any number of them. The other big thing for v0.2 was performance. Having lots of objects around doesn’t hurt performance unless many of them start moving.

The next version (0.3) is mostly about getting the game to the point where content can be produced that won’t need to be thrown away later. A lot of it involves audio and visual features on the client side, so screenshots should start to get prettier.

Anyway, if you have been following this and would like to become an alpha tester, I am still looking for more, so leave a comment below. I still prefer to stick to people I know, but leave a comment if you are interested. The next two weeks will involve a lot of testing.