MV3D Development Blog

Of course, I still feel like there is a mountain of work left to be done. Still got 19 features/bugs to do in v0.2. While 3 of those are in the testing phase, 16 features/bugs is a lot.

Last night we set a new record for number of people on the alpha server. There were 4. Not that many, I know, and I accounted for 3 of those (desktop+2 laptops). However, the server wasn’t even breaking a sweat. It was running at 1-15% CPU usage. I’m guessing, it could support 20ish players in the state it is in now. There is a bit of overhead for each player, and some quick optimizations should be able to fix that. Also, the alpha server is only a P3.

I do wish I knew more about how the big guys run their MMORPGs– server-wise. I mean, I’m planning on a single processor supporting about 100 players. I think that’s a pretty realistic goal, and a dual-quad core server would theoretically support 800 players. Not too shabby. We’ll see if it works. A blade enclosure with 8 blades, each with dual-quad cores could therefore support 6400 players, and a rack full (4 enclosures I think) 25600 players. Cool.. Even at 20 players per processor, that’s still 5120 players for about $120k of hardware. Granted, quad core procs aren’t available just yet, but neither is MV3D.
I’m currently working on connecting multiple areas together. It is mostly working, there is just an annoying bug or 10. Once it works, though, I’ll be able to do some cool tests with several times the world size I have now. It takes about 1-2 minutes to cross the world at this point.

I have been putting off the rest of the redundnacy/failover stuff lately in favor of gateways (connecting areas). I’ll have to get back to that soon though.

The avatars now act fairly realistically… err.. umm.. well, fairly similarly to other games. Not realistic, I suppose, but just what people are used to.

One final thing, a screen shot:

I can’t recall if that is 50 or 100 avatars standing in a line, but it’s still cool either way.

I’m probably not as funny as I think I am. Anyway, before getting into the topic here, I will mention that a lot of the load balancing, redundancy, and fail over is working. Areas and the objects in them all work with it. Next are Realms, Accounts, and Directories. I’m hoping those will be a little easier since they don’t have any moving parts like Areas and Objects. There are still some bugs though. I’ve connected a client to a server that was the slave for an area. Killing the master server makes the client’s view go completely bonkers for about 5-10 seconds before returning to normal. I have to do more research on why that happens. I think disconnecting and reconnecting solves it, so it could be a client issue. Either way, there is more work to be done there.

Now, profiling. Maybe I discussed it earlier. I think I even put up some results. Well, that was using my own home made profiler. Now I got Python’s profiler working with the client and server. After some frustration due to ODE Physics being the cause of all the slowness, I came up with a very good idea. I know it was a good idea because a server with 5000 objects used to run at about 1 iteration every 10 seconds. Now (as long as those objects are stationary), it runs 1 iteration every 0.05 seconds, or 20 per second. In fact, the maximum number of iterations allowed is 20 per second. It only uses 40% CPU.

But what about moving objects, you say? I need to devise a few more tests, but that shouldn’t be a problem as long as you have a reasonable number of them. 1000 or 5000 moving objects in one area is not reasonable. 100-200 is probably maximum (50-160 would be players and the rest, things that just happen to be rolling around or something). But to tell the truth, 5000 objects in one area is a lot. 500-800 maybe (not including terrain, which could be 250 objects, but they don’t have bodies, and they are constantly disabled, so there).

How do I know these things? I don’t really. I have to sit down and figure out what exactly is the expectation for area size. Of course, areas can be any size at all, but physics, I believe, will dictate a maximum complexity. If you are on the 7th plane of hell, and it’s really just a plane with no other geometry, it can be pretty darn big. But if you are in a city with lots of buildings and other objects, smaller may be the way to go.

I am really itching to see how the client performs when you have 5000 objects around. However, I have to rewrite a bit of the code that decides which objects to send to the client. Currently, every 20 iterations or so, it checks all objects (in one big loop) and sends any updates. With 5000 objects, that’ll cause a burp in the server every 20 iterations. Instead, it should do, say, 5 checks per iteration per client.

Meanwhile, I have a tester now. He has been very helpful, though I feel bad since his experience logging in to the server has not been very good. For some reason, he never sees the ground and has a pretty horrible ping time (possibly?). I’m not sure about the ping time yet because there are a few other factors that could cause slow ping like symptoms. For instance, the dreaded bug #100. I still have no clue what’s up with that one. So the odd thing is that you’d think (or I would) that these problems were related to not being on the local LAN with the server. Not true. I’ve logged in a bunch of times today from a T1 line. No problems whatsoever and the lag is minimal. Either way, I am very happy to have a tester, and he is certainly doing a good job and finding all these issues! Well, I guess if he had gotten logged in with no problems, he wouldn’t have much to test.
Speaking of testing. I tested that terrain from below over the T1. It went about the same speed it does over the LAN (actually seemingly faster?). That was a very happy thing. The way terrain works in MV3D is very experimental. Unlike in other games, the client has no preconception of what the terrain is shaped like. It all gets sent from the server to the client in little chunks. The fact that it didn’t seem to slow things down (other than my laptop with its puny 3d card having a bit of trouble rendering it faster than 40fps [which isn’t too shabby, but I get 400 on my desktop]) is definitely a good sign.

Back to more coding.

This picture is humongous, but:

Ooh, pretty. If you are particularly observant, you’d notice that it’s the same landscape as:

That image is from Siegium. I recycled it because it has a nice flat area in the middle to put stuff on. Though I guess I could do that again– all I did was edit the heightfield png image and black out a circle in the center. So yes, I also made a png image to heightfield converter as well.

Anyway, basic terrain is done. It is pretty darn cool because the terrain is broken up into chunks and it uses the PlayerView system just as any other normal object does. That means it is fully deformable (i.e. you can dig a hole in the terrain), there can be multiple layers (for tunnels or overhangs), and it adheres to your view distance. That last one is particularly key. While the server may have 20 sq miles of terrain, the client will only know about (and see) the closest 1/2 mile or so. The next iteration will have grass. Basic sky is done, though I may hack in a sun at some point since that should be quick and should make the terrain look better. By the way, FPS on the client for that image? 300-400ish. Ogre3D, you rock! (I still don’t understand how I can get 400 FPS in a game that is written in an interpreted language [Python])

I’m working on some of the most important features of MV3D right now. Those would be load balancing and server fail-over. Load balancing will make sure that no one game server gets overloaded by too many players/objects/etc, and fail-over is to prevent any downtime in the case of a server exploding (or being upgraded).

So far, things have not been very complicated (admittedly, I’m only doing load balancing/fail over for Areas now, not Objects or Realms or User Accounts, or Directories.) I also haven’t done much testing of the fail-over quite yet. I’m still finishing up unit tests for load balancing/redundancy.

I also ran into a moment of wondering how the heck I was going to load balance / fail over individual objects within areas and almost decided that it wasn’t worth it (and objects should be owned by the area they are in). However, there is a little (and very odd) feature that technically allows objects to be in more than one area at once that makes this not an option. I can’t think of any real uses for this, but I suspect there will be some. It probably doesn’t make much sense for things that are visible, but MV3D’s concept of an Object (or Area) is pretty vague. Some objects will have functions like spawning bad guys or being Gateways between Areas. The former could have a use for being in more than one area now that I think of it. If you wanted to be able to spawn a goblin in one of several areas, an easy way to do it may be to put the spawn object in all of those areas and have it pick one when creating a new goblin.

MV3D also doesn’t care what you put inside of things. So you could easily have a magical bag of holding into which you could chuck several Areas. That would be funny. And wierd. Either way, it’s pretty flexible.

… And I quote: “This document is considerably out of date.”

That makes me remember that I had really wanted to keep up with documentation on MV3D. I haven’t done so in a while. Better look into that.

I’m currently testing some methods to ensure that servers run at 15-20 iterations per second. Since the biggest usage of time seems to be with sending updates out to clients, I’m concentrating there and adding features to adjust the frequency updates are sent out based on various parameters such as client ping time, server load, the distance between the client and the object to update.

I’ve also done a little poking around at other MMORPG source code or documentation (if that wasn’t available). Particularly, I’ve looked at PlaneShift. Some people would probably be unhappy with me for not just contributing to PlaneShift (a free/open source MMORPG that aparently gets 150 concurrent users on a regular basis) and instead starting my own. Well, the reason is that I happen to disagree with a lot of their choices– espeically how their code is structured. If I’m wrong here, PS guys feel free to correct me. If you want to add a new type of sword to the game, not only do you have to restart the server (and drop all the clients), but you have to also re-compile the source code after adding your new object to it. Everything is in the source code. It’s wierd. Oh yeah, they also aparently only envision having one server to support clients, which really severely limits the first M of MMORPG (most would say you lose it completely in that case). There’s also a lot of mucking about with databases in order to make changes to things that are saved there. I don’t have to muck with the database except when I want to wipe it because of a code change or to test things with a clean database. And even that I am planning on automating. Yes, one of my enhancements for v0.2 is to create a web based database explorer that understands the MV3D database scheme, but that’ll just be used for debugging. A normal server operator should only have to create the database & schema, and for the latter, I supply SQL code. That’s it. No more mucking about in the database. The server app should take care of it for you. Besides, for MV3D, the database really isn’t that important. It exists for two reasons:

1. Running in single server mode (so you can save your game environment)
2. When a multi system failure brings down all the primary and secondary servers for an area/object/etc. (So you can reload those areas/objects/etc on another server– or the same one when it comes back, whichever)

So maybe that makes it less impressive that I don’t have to mess with it.

I shouldn’t complain much since PS regularly has 150 players online, and I have been having trouble testing the client with 50. (really! I think it’s the client tester’s issue..)

Anyway, the basic technical vision for MV3D has always been to support any game world you threw at it with any number of players presuming you had enough server hardware to support it.

All right, back to work.

I just finished a ‘Fake Client’ implementation for testing purposes. It’s basically a client that you can give a queue of commands to run and it acts as if it were someone connecting to the server. However, it doesn’t have any graphics or anything, and you can run any number of them at once. I had 50 of them connect to the alpha server (a dual P3, but MV3D doesn’t yet take advantage of both CPUs). The funny part is that the system that ran the clients (a P4) suffered more than the server. It just couldn’t keep up. And the CPU usage was pegged at 100% the whole time. The test took 20-30 minutes to complete, which is kind of sad considering each client did these things:

1. Connected
2. Attached to a character
3. Waited 20 seconds
4. Said “Hello” on chat
5. Waited another 20 seconds
6. Said “Goodbye” on chat
7. Waited 10 seconds
8. Quit

I think a lot of the bottleneck was on the client, but the server was down to 2.5 iterations per second at the worst (at 20 ips, the server runs in real time. Any less, and physics and everything go slower). Without looking at the data, I’d guess that the problem was sending updates to 50 clients for 11 objects up to 20 times per second (11,000 updates per second). As I may have mentioned in an earlier post, sending these updates is probably the one thing that takes up most of the server’s time. A lot of the work I have to do in v0.2 is geared towards lessening the number of updates that are sent. This test should be a good baseline to see how much I improve things.

Almost 9 days of uptime on the alpha server.. And it just took the beating I gave it with no complaints. I’d say we are doing well on the stability front.

Two things on my list for v0.2 are terrain and sky. Those should make the screenshots at least a little more interesting. Terrain is finished but not 100% tested, too..

Either way, here’s a fun shot I took the night before v0.1 was officially finished:

Dude! Get off my head, you’re gonna bend my antenna! For those interested, I took this shot while I was adjusting some physics parameters and because of that, the player could jump high enough to land on another player’s head.

This report is running short on un-fixed bugs. Right now, there are 0. I’ve tested on a few windows systems and of course my linux system. I’m making an attempt at installing Max OSX under VMWare, but we’ll see how far I get since I’d likely only be able to get everything compiled rather than actually running it (running 3d apps under VMWare suxors).

There were some annoying bugs such as one that took all of yesterday to get working, but the server has been incredibly stable. The alpha test server has been running for almost 5 days now and I’ve been abusing it as much as I can. I am especially pleased with error handling on the server. In the course of fixing bugs, I was connecting with buggy clients and clients with somewhat different ideas of what the server should be doing and even when a Python error was generated on the server, it kept on ticking. In fact due to a bug with player movement yesterday, when I connect to the server now, the server and client have a completely different idea of how the player moves and so the player’s body gets put half way into the ground. It’s not a bug because the server code running on the alpha server is 5 days old. It works fine connecting to a more recent version of the server. Just that it’s impressive that the older version still works even if it is funny looking.

I did end up sneaking in a little feature yesterday. It was necessary because of the bug I was fixing, honest. I can now do some pretty cool ray-casting on the server. I had been very worried about whether this would work or not and I had no clue how to do it without a graphics engine. The server doesn’t have any concept of graphics, just physics. Anyway, I use it to ‘float’ the player’s body off the ground so that it doesn’t drag it’s ass across the floor. Player bodies are now represented by a pill shaped object for physics. Really, the previous way of doing player physics wouldn’t have coped with uneven ground surfaces. Stairs would have been completely out of the question. Before, the player was a pill shape balanced on a sphere that would roll around the ground.

I’m definitely itching to do some more features, but I’ve decided that I at least need a week of bug hunting. So, assuming there aren’t any other major bugs, the feature freeze should be lifted Tuesday. I’ll tag that code as version 0.1 and move on to 0.2.

The versioning scheme I’m thinking of going with is pretty stright forward. v0.05 was the pre-alpha version. As soon as I completed the features and once all the bugs are squashed, v0.1 is done. That version has the very core features in it such as networking integration, physics integration, client with 3d engine, and basic support of multiple servers. More or less 0.1 encompassed a basic implimentation of all core features. v0.2 will have full functionality for all core network/server related features. v0.3 will have full functionality for visual and content related features. v0.4 should see higher level features and features specific to the beta game worlds such as NPC pathfinding, player abilities, quests, and game editor tools. After that is 0.5, or the official beginning of the beta test. The beta test is meant to be a feature complete test with partial content (game world stuff). Content will keep being generated through the beta test and at v0.7, it’ll be a content complete betatest. That’ll mean all content for the 1.0 release has been completed and is ready for testing. The next step after that is 1.0. There’s a lot to do between now and then, but I feel that my decision to pay a lot of attention to the lower level core functionality is really helping things along.

That’s where I am now. I have more screenshots, but they are just robots and a simple ground plane like the other one, so not too interesting.

So far, the maximum connect count to a MV3D server is 3 clients at the same time. It may not seem like much, but that’ 3 times the amount that worked with the old code. I’ve also done the first test of a remote client that wasn’t on the same network as the server. It was fine. Though I decided to do a low bandwidth test where I simulated a very low bandwidth/laggy client and that wasn’t so fine. But to MV3D’s credit, in the ssh window I had open to the server, if I typed a letter, maybe it would get echoed back in 2 seconds. So this was probably <1kb/s connection, and that is very slow. The problem was since I’m using TCP (instead of UDP), all data has to get to the other end of the connection and it has to get there in order. So, the server was sending out update after update (20x per object per second) and the client would get them all and apply them in order. There are a couple of ways to fix this. One is to update less often (and base the frequency of updates on the saturation of the user’s connection [or a best guess thereof]), or to start using UDP for some things. Both may be a good idea, though I’ll have to see if PB works over UDP. Anyway, the slowness was not really unexpected.

Meanwhile, I’m finishing up moving all the source code files around and after that, the Alpha Test will start. I have 2-3 files left to move, so tomorrow’s probably the day. When I started the project, I wasn’t sure how I wanted to organize the code, so I put most of everything into one big directory figuring I’d find a good directory structure and move it in rather than making a directory structure that turned out to be retarded later. Because I’m using Subversion and not CVS, this has not been too big a task. Just have to move each file then update all the references to it. One by one.

MV3D took a pretty big step forward last night. I ended up working on the problem with player movement and multiple servers. That means I setup two seperate MV3D servers that would share the simulation of the game world between them. In this case, I was simulating a simulation server which would live in the middle part of the network and a player server to which players could directly log in. (Standard server setup for those not in the know– a front end server interfaces with the public (through a firewall). The front end server talks through another firewall to a mid level server (sometimes called application server). The front level server is more or less a proxy whereas the mid level server does a lot of the calculations. The mid level server(s) talk to back end servers (database servers and such) through a third firewall (if you believe Cisco– and keep in mind they’d like to sell you all three firewalls for about the cost of a nice car).

Aaaaanyway, I was using a front end player server and mid level simulation/realm server. So the player connects to the player server which then gets info from the simulation server about where objects are and what they’re doing. There were quite a few bugs.. In things I had written unit tests for even. But it works now with one sort of annoying bug.

It was a real testament to Twisted (and me of course! hah) that connecting through the player server to the sim server (running on a seperate machine) didn’t incur any real noticeable lag. Granted, the two servers were on the same network, but that would normally be the case in a real production level MV3D environment. (in fact, they’d be on a gigabit+ network, none of this silly 100mbit stuff that I have)

Either way, it was really cool (even though it did exactly the same thing that running one server does) because it shows that the multiple server design of MV3D is promising. (and at least mostly functional)

The player server didn’t even have the username and password for the player I logged in as. It got that off the other server as well (and still no noticeable lag).

Any alpha testers? I’ve decided that I don’t want to open up the test to anyone that I don’t know. I should be able to finish the remaining items listed as enhancements for the alpha this weekend.

We’re going to Alpha at least. According to my spiffy new system, I have 2 enhancements and 2 defects to complete before I can officially say the alpha test is ready. Namely: add god-mode world editing, finish game state persisting to database, fix problem where colliding with other players causes viewpoint instability, and fix player movement in a multiple server environment. The last one will be fun to test because I’ll be hooking up a client for the first time into a MV3D game that is running on more than one server. In fact, there hasn’t been a game running on multiple servers yet. The functionality has been tested in Unit Tests and works fine, but I haven’t run two servers for long periods of time like that.

Speaking of “spiffy new system,” you can check it out at:
http://dev.mv3d.com:8080/cgi-bin/trac.cgi/wiki

I got all of character generation (for the alpha test) done last night. That and persisting are the only major hurdles I believe. God mode editing is planned out and ready for programming.

One of the next big hurdles coming up after the brunt of the bugs found in the alpha test are squashed is terrains. It’s hard to have a MMORPG without ground. Since Ogre doesn’t support terrain in any sort of dynamic way, it’s going to be up to me. I am going to try to generate meshes on the fly for patches of terrain which will correspond directly to some objects on the server side. In fact, some of the mesh generation algorithms can be shared between the two since meshes will be required within ODE for collision. ODE does have a terrain collision model, but I think it’s still in the experimental stage and is not part of the main ODE distribution. Eventually, we will switch to that. Anyway, the size and detail of these terrain pieces is going to be up for debate (um.. I’ll be debating with myself) since how well they perform will depend on getting the right size/detail. For those in the know, I should answer the question as to why I’m not going with some sort of progressive mesh for these like CLOD or whatever. The reason is that I’ve seen a few places that are saying that with modern graphics cards and a modern 3d engine, progressive meshes actually are slower than just bruteforce rendering of the terrain. We’ll put that theory to the test. And I fully intend to do a significant terrain load test on both the server and the client. I know ODE can handle it– it did just fine with fairly large terrains in Siegium. Can Ogre handle it? Dunno. (and can Python generate these meshes fast enough?)

That’s probably all for now.

So after some frustrations with getting the physics engine to cooperate, there is now some semblance of normal player movement. You can walk, and turn and when you bump into things, you push them instead of falling over. In fact, you don’t fall over (which is good, and making it work like that was the source of a lot of frustration). The client does get a little wonky when you collide into another player at high speed (need to limit the walking speed still), but if you disconnect and reconnect, it’s normal again.

At this point, you can run the client (still only on linux) then enter your host name, login, and password. Click connect and (if you didn’t mess up the login info) it’ll let you select a character to connect to (since one account can have multiple characters). The server creates any number of characters (I have it set to 10 for now) and it creates a single account that can connect to all of them. Select a character and click “Connect” and you will start playing as that character. You get to see out of the character’s eyes and steer it around with the arrow keys.

The server running on NightEyes was up for 4 days before I took it down. And I only took it down to upgrade it to the newest version. Connecting to it (rather than a server hosted on my desktop machine) doesn’t give any noticeable amount of latency either! So this is good. The question is just what’s the latency connecting to it over the internet? It may be very laggy since I think it updates the position of every moving object 10x per second. In any case, the new version of the server isn’t taking any more CPU time than the old one, so that’s good.

I don’t believe I have that many more things that I need to do for the Alpha Test to start.According to the list, I think I just need to add Chat, In Game Editing, make sure that saving the game world to DB really works, and then setup the Asset downloading mechanism. Oh yeah, and the most important but probably most painful– get it working on Windows. Anyway, I think this means I’m ready to ask the question:

Who wants to be an Alpha Tester? Really, at this point, you will be a QA person. Your mission will be to break things on the server & client and report bugs. The game is not fun. It’s not even a game at this point. I would love to get up to 5 people to start. It would be nice to have people who run on several OS’s (Windows, Mac, Linux). However, Linux people who have 3D cards and know their way around a command line will be the first I’ll need. I actually suspect that the game will work on Mac out of the box-ish, but whoever volunteers for that will need to compile and install things on their system since I won’t be able to produce binaries. So here’s the requirements:

• Fairly decent (P4) Linux, Windows, or Mac system
• 3D Card bought in the past 5 years
• For Mac and Linux, some programming experience (or at least knowledge of how to compile and install software)
• A lot of patience

And here’s what you’ll have to do:

• List your system specs
• Install the software according to instructions
• Run the Unit Tests for the software on your system and report results
• Run the client and report any bugs
• There will probably be several server load tests where we will need to arrange a time for all testers to connect to the server
• Submit bug reports when you find them
• Help narrowing down bugs to determine what the smallest set of actions is to cause them

Any volunteers? I’m only looking for people who are serious about testing. I suspect the test will start officially in a few weeks, but Linux (and maybe Mac) people can probably get in on it earlier.

I may need to turn people down as testers if I get too many (or too many on one OS or whatever). I’ll also be looking at people I know first. Don’t be disappointed. As things progress, I’ll let more and more testers in. Also please forgive me if it’s more than two weeks until the test starts, or if you have an insanely hard time getting the software to work on your system. This is a very early stage test, so there’s going to be lots of bugs and strange problems.

I almost forgot my screenshot!