MV3D Development Blog

Yaaaaay!

Anyway. v0.2 is done, v0.3 is in progress. (yaaay) Man, I’ve got to get some screen shots because character animation, lights, shadows, and water came together really quick. Now water isn’t finished because it isn’t integrated into the physics as I’d like it to be. Even with the Example Ogre water shaders, it looks cool. Oh yeah, so water physics. I think I’m going to try something, but I don’t know if it’ll work. My idea is for bigger bodies of water to define their shape in the physics engine, but instead of having things bounce off of them, a) let whatever comes in contact with it know that it has touched water (so if say you drop a sword into water, it could rust), but more importantly, b) slow the object down and do some very cheezy fake bouyancy tricks. So if you put something in water that is less dense than water, it floats, but otherwise, it sinks. Very fake and simple, but something. Could be cool, could suck, could be too CPU intensive. Should be easy to program.

What I’m working on now are the two more complex problems in v0.3. The one that I would say is mostly done is allowing each area in the game to have its own coordinate system while keeping a seamless transition from one area to the next both physically and visually. It does work at the moment. There is one physical glitch where if you stand right between the two areas, you seem to fall through the ground for some reason, and one visual glitch where there are a couple of frames of discombobulation when passing through the areas. The former should be solveable once I figure out why it’s happening– I am pretty sure I took that problem into account when designing how this all works. The latter may be a little tougher. It’s sort of an order issue. Visual representations are now grouped together in Ogre by area (before everything was all together on the client side), and whole areas can be slid around. Ok, so I didn’t want to get into this, but I think I have to in order to explain fully. Areas are seperated by Gateways, and Gateways can now define a translation and a rotation that happens to objects as they pass through (or see through) the gateway. The translation and rotation specify how to change a given point from Area a’s coordinate system into area b’s. So the server looks through the player’s eyes and finds all the areas they can see and determines that area a is where the player is, so that has a translation of (0,0,0), area b is through a gateway with a (100,0,0) translation, so it’s at (100,0,0). Area c is through a gateway in area b with a (100,0,0) translation, so it is at (200,0,0). It sends all this to the client, and the client (having grouped all of the objects for a given area together) moves them all into their respective offsets.

The problem comes in when you move from area a to b, the server will add (100,0,0) to the PC’s position and send notice of the area change to the PC. When the client gets the position change, it updates accordingly. When it gets the area change, it has to go through all the areas it knows about and adjust their offsets by (-100,0,0). It could just wait for the server to send out offsets again, but the server only does that every few seconds. Anyway, there’s a few frames between changing the position and changing the offsets where either your body is missing, or your surroundings are missing because one or the other hasn’t been updated. Fun stuff. I’ll figure it out. For the moment, I’m living with it and moving on to the other complicated v0.3 thing:

Creating a type of Area that can automatically split itself up amongst several servers. And you thought I hashed load balancing to death in the last update? Oh no! There’s more. Currently, the lowest level that can be load balanced is a whole Area. Under most circumstances, this is perfectly fine. But something that tends to happen in MMOGs is that certain areas/zones become very popular and you get many people hanging around in them. Often, this causes servers or clients to bog down. For now, I don’t care about the client part, but the server part is important. What do you do? Well, you make an area that can distribute pieces of itself across servers. So I am working on an octree type thing that’ll do just this. Somehow, I’ve managed to avoid octrees and quadtrees in my previous programming experience, but they seem pretty neat and are pretty much the way to go for what I’m trying to do. [an octree is a tree structure where each branch has either 8 sub branches or some number of leaves (not both)] The hard part comes in with distributing things. I suspect it’ll be a matter of finding a quick and painless way to change the master server for groups of objects at a time. Currently, if you make a master server into a slave server for an object, it dumps the object and reloads it over the network from the new master server. Not too fast. Hopefully, I can keep the object and somehow reconnect it to the new server.

Finally, I had a very good lunch with and his coworker today. Funnily enough, we compared stories of how slow / annoying our individual methods of persisting random Python objects to database without requiring a specific schema were. We chatted about a lot of other things of course, some of which I may bring up here in the future if they come to fruition. But in any case, it was good to talk to other programmer types about MMO related things.

And finally finally, MV3D dev may slow down a bit over the next month because I’ll be working on two video jobs.

Some cool things have happened in the last few days. The big one is that I was able to successfully have an avatar travel from one area (hosted by server A) into another area (hosted by server B). It is completely seamless as well. That is definitely a big step, and it took quite a bit of bugfixing to get working. Basically, it means that the game world can now extend further than the area a single server can host. I was also able to connect to a server with 1000 other avatars standing around. Here is a pretty cool screenshot:

I had to jump up a bit to get a good view. That’s not all 1000; there were several hundred on the other side of the hill as well. The ones are bunched up in the lower left because (I think) they fell down the hill over there. I really have to get rid of the robots though. I will do so after I get them to walk properly (display a walking animation as they move). That’s one of the first things I’ll probably hit on in the next version.

Speaking of the next version. I still have one open bug for this version, but I can’t reproduce it consistently. Sometimes Python prints out an error when you exit the client. That’s all. It’s pretty low priority as it doesn’t really cause any actual problems. I tried a fix, but since I can’t reproduce it on command, I think if it doesn’t happen in the next couple of days, I’ll say v0.2 is done and move on to v0.3.

Here’s another screen shot with 500 avatars and 1000 blocks:

Things seem pretty stable now. As part of the Test Plan, I’ve tested more of the redundancy stuff with a client connected. Now, you don’t really notice losing a server (except the one the client is directly connected to, but that’s a v0.3 feature ). I even tested whacking a login server without issues– it just finds another (automatically even).

I feel like I’ve fixed a million bugs in the last few days.. There were certainly a lot of them. But everything seems pretty good now. I am definitely looking forward to v0.3, though v0.4 is going to be the real fun one. That’s when it starts to become a game.

One more thing. The features that are in place now are almost all of the features a simulation server needs to have. There are a couple more scheduled for the next version or two, most notably: allowing areas to have their own coordinate systems, and breaking a congested area up into pieces to allow multiple servers to split processing of the area between them. Everything else is not strictly simulation based, so it can run on seperate dedicated servers if needs be. This seems to say that Python will work for the server, and that in general, MV3D is very possibly going to work. The foundation of the game is coded and turned out to be very stable. Now that’s a milestone!

Or as B&B put it, “Testies, testies, 1.. 2… 3?”

I’ve got a couple of client related bugs to fix, but then I’ll probably send out the second test version. Maybe a few days, though one bug is definitely a wierd one. All of a sudden, the client started crashing whenever it was starting to display the game world. Other than UI related things and stack thrashing, crashes have been incredibly rare since I’m using Python. I traced this one to the import statement that loads the code to handle the client side of the player’s avatar. Putting a statement before it that prints a test message to the window and does nothing more fixes the problem. I’ve seen this sort of thing many times before in multi-threaded applications. The problem is that the MV3D client isn’t multi-threaded. It makes no sense for a problem like this to occur in a single-threaded app, but well, got to figure that one out.

I was able to increase the server performance by a factor of 10, though. Because of the floating point precision issues I mentioned below, ODE physics does not like big things. I knew when I started using it that the robot model was frikkin’ huge, and I thought maybe it may mess up ODE, but I didn’t want to screw around with scaling in Ogre. Anyway, scaling things by 1/10 allowed me to increase the amount of time that the physics engine could move the simulation ahead each iteration and therefore, went from having to run physics 10 times per iteration to having to do it 1 time. Since collision detection in the physics engine was the reason things were going slow, this bumped the speed up by at least 10 times. In fact, the following screen shot shows the client connected to a world with 2410 objects in it:

I haven’t scaled the landscape down to 1/10 scale yet so it’s blocky, but that’s next on the agenda. I should actually go even smaller because I like using 1 meter = 1 game unit. The robot is 8 game units high. He should probably be more like 2-3. Really, I should ditch the Robot. Soon, oh yes. Soon.

I did play with freeze a bit yesterday. I was able to freeze the client app. Freeze crunches up a Python program in to a Unix executable ELF file. I still have to play around because freeze doesn’t include any pyd/so files.. It also doesn’t know about any things that are imported based on what the server tells it to import. I suspect I can do what I do on the Windows version, which is include the source trees for Client and Util so that the executable can just import out of there. No clue if that works for freeze, but I hope so. It may be hard for me to test as well since I don’t have a second Linux box that has a 3d graphics card in it. I may have to blindly ship it out to a tester unfortunately because any results that I get on my desktop could be invalid. I have all the libraries and python stuff already installed, so the app may work fine on my machine, but fail miserably elsewhere.

I’ve also been thinking about ideas for the next version. In particular, terrain features and in game editing. For the terrain, I have to find a good way to allow for things like roads, paths, and rocky vs grassy surfaces. Right now, each large terrain chunk must have a single texture. What would be cool would be to have a base texture and then be able to overlay other textures on top of it in such a way that they blended at the edges. Then of course, on the terrain comes grass/flowers/small bushes/etc. There is already a sample PyOgre program that makes grass, so I’ll use that.. Or just make a generic way to put stuff on the terrain.

The In Game Editor, which I am creatively calling IGE, is going to be a fun one. It has to be very easy to use, very powerful, and very flexible. A tough combo, certainly. Good UIs are not my strongpoint (even though that was the brunt of my first professional programming job). This is going to require a lot of thinking. If anyone who reads this has any suggestions for how they’d like an in game MORPG world editor to work, please comment.
Finally, if you haven’t seen it already, MMORPG fans or gamers in general will get a kick out of this South Park episode from a few weeks ago. Make Love, not Warcraft. So funny.

V0.2 officially enters the testing phase as soon as I verify that one bug is fixed and fix a second. Yes, I got all of the rest of the features done except for one hat I decided needed more thought, so I postponed it until the next version. The problem mainly was that in order to figure out how the feature should work, things needed to be a little further along then they are. Anyway, I’m excited about the new version. On the surface, it doesn’t really seem like a lot has been added. The client now has terrain and a sky, but nothing much else. Under the hood, though, a lot of things are there that hadn’t been including all the systems for connecting servers together to distribute the game.

Speaking of that, in my last post, I mentioned that I wish I knew how the other MMORPGs did it. Since then, I was able to at least partially answer that question. This white-paper is a very long read, but anyone interested in MMORPG development should consider it. It is a document put together by some of the bigger guys in the MMORPG field which goes over many aspects of creating MMORPGs. It was fairly informative, though a lot of the stuff I already knew or had guessed. It sounds like my method of having lots of little servers simulate the game world is the most flexible way of doing things (although not the easiest). The easiest of course is to throw some big iron at the problem and get a 64/128 CPU server to handle your whole game. That’s a little out of my budget, and when that big honking machine crashes, down goes your game. In my case, it would take quite a few servers going down for clients to see downtime. If one server goes down, the load balancing automatically replaces it. How cool is that? And I’m not talking pie in the sky features that aren’t coded, I’m talking currently finished and tested features. Well, they have to go through the release testing, but they’ve already been internally tested by me. The load balancing works at pretty much all levels. The very top level servers, though, are redundant and have fail over, but can not appoint new top level servers. Fortunately, they work off of a database (or replicated databases), so there can be any number of them. The other big thing for v0.2 was performance. Having lots of objects around doesn’t hurt performance unless many of them start moving.

The next version (0.3) is mostly about getting the game to the point where content can be produced that won’t need to be thrown away later. A lot of it involves audio and visual features on the client side, so screenshots should start to get prettier.

Anyway, if you have been following this and would like to become an alpha tester, I am still looking for more, so leave a comment below. I still prefer to stick to people I know, but leave a comment if you are interested. The next two weeks will involve a lot of testing.

Of course, I still feel like there is a mountain of work left to be done. Still got 19 features/bugs to do in v0.2. While 3 of those are in the testing phase, 16 features/bugs is a lot.

Last night we set a new record for number of people on the alpha server. There were 4. Not that many, I know, and I accounted for 3 of those (desktop+2 laptops). However, the server wasn’t even breaking a sweat. It was running at 1-15% CPU usage. I’m guessing, it could support 20ish players in the state it is in now. There is a bit of overhead for each player, and some quick optimizations should be able to fix that. Also, the alpha server is only a P3.

I do wish I knew more about how the big guys run their MMORPGs– server-wise. I mean, I’m planning on a single processor supporting about 100 players. I think that’s a pretty realistic goal, and a dual-quad core server would theoretically support 800 players. Not too shabby. We’ll see if it works. A blade enclosure with 8 blades, each with dual-quad cores could therefore support 6400 players, and a rack full (4 enclosures I think) 25600 players. Cool.. Even at 20 players per processor, that’s still 5120 players for about $120k of hardware. Granted, quad core procs aren’t available just yet, but neither is MV3D.
I’m currently working on connecting multiple areas together. It is mostly working, there is just an annoying bug or 10. Once it works, though, I’ll be able to do some cool tests with several times the world size I have now. It takes about 1-2 minutes to cross the world at this point.

I have been putting off the rest of the redundnacy/failover stuff lately in favor of gateways (connecting areas). I’ll have to get back to that soon though.

The avatars now act fairly realistically… err.. umm.. well, fairly similarly to other games. Not realistic, I suppose, but just what people are used to.

One final thing, a screen shot:

I can’t recall if that is 50 or 100 avatars standing in a line, but it’s still cool either way.

I’m probably not as funny as I think I am. Anyway, before getting into the topic here, I will mention that a lot of the load balancing, redundancy, and fail over is working. Areas and the objects in them all work with it. Next are Realms, Accounts, and Directories. I’m hoping those will be a little easier since they don’t have any moving parts like Areas and Objects. There are still some bugs though. I’ve connected a client to a server that was the slave for an area. Killing the master server makes the client’s view go completely bonkers for about 5-10 seconds before returning to normal. I have to do more research on why that happens. I think disconnecting and reconnecting solves it, so it could be a client issue. Either way, there is more work to be done there.

Now, profiling. Maybe I discussed it earlier. I think I even put up some results. Well, that was using my own home made profiler. Now I got Python’s profiler working with the client and server. After some frustration due to ODE Physics being the cause of all the slowness, I came up with a very good idea. I know it was a good idea because a server with 5000 objects used to run at about 1 iteration every 10 seconds. Now (as long as those objects are stationary), it runs 1 iteration every 0.05 seconds, or 20 per second. In fact, the maximum number of iterations allowed is 20 per second. It only uses 40% CPU.

But what about moving objects, you say? I need to devise a few more tests, but that shouldn’t be a problem as long as you have a reasonable number of them. 1000 or 5000 moving objects in one area is not reasonable. 100-200 is probably maximum (50-160 would be players and the rest, things that just happen to be rolling around or something). But to tell the truth, 5000 objects in one area is a lot. 500-800 maybe (not including terrain, which could be 250 objects, but they don’t have bodies, and they are constantly disabled, so there).

How do I know these things? I don’t really. I have to sit down and figure out what exactly is the expectation for area size. Of course, areas can be any size at all, but physics, I believe, will dictate a maximum complexity. If you are on the 7th plane of hell, and it’s really just a plane with no other geometry, it can be pretty darn big. But if you are in a city with lots of buildings and other objects, smaller may be the way to go.

I am really itching to see how the client performs when you have 5000 objects around. However, I have to rewrite a bit of the code that decides which objects to send to the client. Currently, every 20 iterations or so, it checks all objects (in one big loop) and sends any updates. With 5000 objects, that’ll cause a burp in the server every 20 iterations. Instead, it should do, say, 5 checks per iteration per client.

Meanwhile, I have a tester now. He has been very helpful, though I feel bad since his experience logging in to the server has not been very good. For some reason, he never sees the ground and has a pretty horrible ping time (possibly?). I’m not sure about the ping time yet because there are a few other factors that could cause slow ping like symptoms. For instance, the dreaded bug #100. I still have no clue what’s up with that one. So the odd thing is that you’d think (or I would) that these problems were related to not being on the local LAN with the server. Not true. I’ve logged in a bunch of times today from a T1 line. No problems whatsoever and the lag is minimal. Either way, I am very happy to have a tester, and he is certainly doing a good job and finding all these issues! Well, I guess if he had gotten logged in with no problems, he wouldn’t have much to test.
Speaking of testing. I tested that terrain from below over the T1. It went about the same speed it does over the LAN (actually seemingly faster?). That was a very happy thing. The way terrain works in MV3D is very experimental. Unlike in other games, the client has no preconception of what the terrain is shaped like. It all gets sent from the server to the client in little chunks. The fact that it didn’t seem to slow things down (other than my laptop with its puny 3d card having a bit of trouble rendering it faster than 40fps [which isn’t too shabby, but I get 400 on my desktop]) is definitely a good sign.

Back to more coding.