MV3D Development Blog

This picture is humongous, but:

Ooh, pretty. If you are particularly observant, you’d notice that it’s the same landscape as:

That image is from Siegium. I recycled it because it has a nice flat area in the middle to put stuff on. Though I guess I could do that again– all I did was edit the heightfield png image and black out a circle in the center. So yes, I also made a png image to heightfield converter as well.

Anyway, basic terrain is done. It is pretty darn cool because the terrain is broken up into chunks and it uses the PlayerView system just as any other normal object does. That means it is fully deformable (i.e. you can dig a hole in the terrain), there can be multiple layers (for tunnels or overhangs), and it adheres to your view distance. That last one is particularly key. While the server may have 20 sq miles of terrain, the client will only know about (and see) the closest 1/2 mile or so. The next iteration will have grass. Basic sky is done, though I may hack in a sun at some point since that should be quick and should make the terrain look better. By the way, FPS on the client for that image? 300-400ish. Ogre3D, you rock! (I still don’t understand how I can get 400 FPS in a game that is written in an interpreted language [Python])

I’m working on some of the most important features of MV3D right now. Those would be load balancing and server fail-over. Load balancing will make sure that no one game server gets overloaded by too many players/objects/etc, and fail-over is to prevent any downtime in the case of a server exploding (or being upgraded).

So far, things have not been very complicated (admittedly, I’m only doing load balancing/fail over for Areas now, not Objects or Realms or User Accounts, or Directories.) I also haven’t done much testing of the fail-over quite yet. I’m still finishing up unit tests for load balancing/redundancy.

I also ran into a moment of wondering how the heck I was going to load balance / fail over individual objects within areas and almost decided that it wasn’t worth it (and objects should be owned by the area they are in). However, there is a little (and very odd) feature that technically allows objects to be in more than one area at once that makes this not an option. I can’t think of any real uses for this, but I suspect there will be some. It probably doesn’t make much sense for things that are visible, but MV3D’s concept of an Object (or Area) is pretty vague. Some objects will have functions like spawning bad guys or being Gateways between Areas. The former could have a use for being in more than one area now that I think of it. If you wanted to be able to spawn a goblin in one of several areas, an easy way to do it may be to put the spawn object in all of those areas and have it pick one when creating a new goblin.

MV3D also doesn’t care what you put inside of things. So you could easily have a magical bag of holding into which you could chuck several Areas. That would be funny. And wierd. Either way, it’s pretty flexible.

… And I quote: “This document is considerably out of date.”

That makes me remember that I had really wanted to keep up with documentation on MV3D. I haven’t done so in a while. Better look into that.

I’m currently testing some methods to ensure that servers run at 15-20 iterations per second. Since the biggest usage of time seems to be with sending updates out to clients, I’m concentrating there and adding features to adjust the frequency updates are sent out based on various parameters such as client ping time, server load, the distance between the client and the object to update.

I’ve also done a little poking around at other MMORPG source code or documentation (if that wasn’t available). Particularly, I’ve looked at PlaneShift. Some people would probably be unhappy with me for not just contributing to PlaneShift (a free/open source MMORPG that aparently gets 150 concurrent users on a regular basis) and instead starting my own. Well, the reason is that I happen to disagree with a lot of their choices– espeically how their code is structured. If I’m wrong here, PS guys feel free to correct me. If you want to add a new type of sword to the game, not only do you have to restart the server (and drop all the clients), but you have to also re-compile the source code after adding your new object to it. Everything is in the source code. It’s wierd. Oh yeah, they also aparently only envision having one server to support clients, which really severely limits the first M of MMORPG (most would say you lose it completely in that case). There’s also a lot of mucking about with databases in order to make changes to things that are saved there. I don’t have to muck with the database except when I want to wipe it because of a code change or to test things with a clean database. And even that I am planning on automating. Yes, one of my enhancements for v0.2 is to create a web based database explorer that understands the MV3D database scheme, but that’ll just be used for debugging. A normal server operator should only have to create the database & schema, and for the latter, I supply SQL code. That’s it. No more mucking about in the database. The server app should take care of it for you. Besides, for MV3D, the database really isn’t that important. It exists for two reasons:

1. Running in single server mode (so you can save your game environment)
2. When a multi system failure brings down all the primary and secondary servers for an area/object/etc. (So you can reload those areas/objects/etc on another server– or the same one when it comes back, whichever)

So maybe that makes it less impressive that I don’t have to mess with it.

I shouldn’t complain much since PS regularly has 150 players online, and I have been having trouble testing the client with 50. (really! I think it’s the client tester’s issue..)

Anyway, the basic technical vision for MV3D has always been to support any game world you threw at it with any number of players presuming you had enough server hardware to support it.

All right, back to work.

I just finished a ‘Fake Client’ implementation for testing purposes. It’s basically a client that you can give a queue of commands to run and it acts as if it were someone connecting to the server. However, it doesn’t have any graphics or anything, and you can run any number of them at once. I had 50 of them connect to the alpha server (a dual P3, but MV3D doesn’t yet take advantage of both CPUs). The funny part is that the system that ran the clients (a P4) suffered more than the server. It just couldn’t keep up. And the CPU usage was pegged at 100% the whole time. The test took 20-30 minutes to complete, which is kind of sad considering each client did these things:

1. Connected
2. Attached to a character
3. Waited 20 seconds
4. Said “Hello” on chat
5. Waited another 20 seconds
6. Said “Goodbye” on chat
7. Waited 10 seconds
8. Quit

I think a lot of the bottleneck was on the client, but the server was down to 2.5 iterations per second at the worst (at 20 ips, the server runs in real time. Any less, and physics and everything go slower). Without looking at the data, I’d guess that the problem was sending updates to 50 clients for 11 objects up to 20 times per second (11,000 updates per second). As I may have mentioned in an earlier post, sending these updates is probably the one thing that takes up most of the server’s time. A lot of the work I have to do in v0.2 is geared towards lessening the number of updates that are sent. This test should be a good baseline to see how much I improve things.

Almost 9 days of uptime on the alpha server.. And it just took the beating I gave it with no complaints. I’d say we are doing well on the stability front.

Two things on my list for v0.2 are terrain and sky. Those should make the screenshots at least a little more interesting. Terrain is finished but not 100% tested, too..

Either way, here’s a fun shot I took the night before v0.1 was officially finished:

Dude! Get off my head, you’re gonna bend my antenna! For those interested, I took this shot while I was adjusting some physics parameters and because of that, the player could jump high enough to land on another player’s head.

This report is running short on un-fixed bugs. Right now, there are 0. I’ve tested on a few windows systems and of course my linux system. I’m making an attempt at installing Max OSX under VMWare, but we’ll see how far I get since I’d likely only be able to get everything compiled rather than actually running it (running 3d apps under VMWare suxors).

There were some annoying bugs such as one that took all of yesterday to get working, but the server has been incredibly stable. The alpha test server has been running for almost 5 days now and I’ve been abusing it as much as I can. I am especially pleased with error handling on the server. In the course of fixing bugs, I was connecting with buggy clients and clients with somewhat different ideas of what the server should be doing and even when a Python error was generated on the server, it kept on ticking. In fact due to a bug with player movement yesterday, when I connect to the server now, the server and client have a completely different idea of how the player moves and so the player’s body gets put half way into the ground. It’s not a bug because the server code running on the alpha server is 5 days old. It works fine connecting to a more recent version of the server. Just that it’s impressive that the older version still works even if it is funny looking.

I did end up sneaking in a little feature yesterday. It was necessary because of the bug I was fixing, honest. I can now do some pretty cool ray-casting on the server. I had been very worried about whether this would work or not and I had no clue how to do it without a graphics engine. The server doesn’t have any concept of graphics, just physics. Anyway, I use it to ‘float’ the player’s body off the ground so that it doesn’t drag it’s ass across the floor. Player bodies are now represented by a pill shaped object for physics. Really, the previous way of doing player physics wouldn’t have coped with uneven ground surfaces. Stairs would have been completely out of the question. Before, the player was a pill shape balanced on a sphere that would roll around the ground.

I’m definitely itching to do some more features, but I’ve decided that I at least need a week of bug hunting. So, assuming there aren’t any other major bugs, the feature freeze should be lifted Tuesday. I’ll tag that code as version 0.1 and move on to 0.2.

The versioning scheme I’m thinking of going with is pretty stright forward. v0.05 was the pre-alpha version. As soon as I completed the features and once all the bugs are squashed, v0.1 is done. That version has the very core features in it such as networking integration, physics integration, client with 3d engine, and basic support of multiple servers. More or less 0.1 encompassed a basic implimentation of all core features. v0.2 will have full functionality for all core network/server related features. v0.3 will have full functionality for visual and content related features. v0.4 should see higher level features and features specific to the beta game worlds such as NPC pathfinding, player abilities, quests, and game editor tools. After that is 0.5, or the official beginning of the beta test. The beta test is meant to be a feature complete test with partial content (game world stuff). Content will keep being generated through the beta test and at v0.7, it’ll be a content complete betatest. That’ll mean all content for the 1.0 release has been completed and is ready for testing. The next step after that is 1.0. There’s a lot to do between now and then, but I feel that my decision to pay a lot of attention to the lower level core functionality is really helping things along.

That’s where I am now. I have more screenshots, but they are just robots and a simple ground plane like the other one, so not too interesting.

So far, the maximum connect count to a MV3D server is 3 clients at the same time. It may not seem like much, but that’ 3 times the amount that worked with the old code. I’ve also done the first test of a remote client that wasn’t on the same network as the server. It was fine. Though I decided to do a low bandwidth test where I simulated a very low bandwidth/laggy client and that wasn’t so fine. But to MV3D’s credit, in the ssh window I had open to the server, if I typed a letter, maybe it would get echoed back in 2 seconds. So this was probably <1kb/s connection, and that is very slow. The problem was since I’m using TCP (instead of UDP), all data has to get to the other end of the connection and it has to get there in order. So, the server was sending out update after update (20x per object per second) and the client would get them all and apply them in order. There are a couple of ways to fix this. One is to update less often (and base the frequency of updates on the saturation of the user’s connection [or a best guess thereof]), or to start using UDP for some things. Both may be a good idea, though I’ll have to see if PB works over UDP. Anyway, the slowness was not really unexpected.

Meanwhile, I’m finishing up moving all the source code files around and after that, the Alpha Test will start. I have 2-3 files left to move, so tomorrow’s probably the day. When I started the project, I wasn’t sure how I wanted to organize the code, so I put most of everything into one big directory figuring I’d find a good directory structure and move it in rather than making a directory structure that turned out to be retarded later. Because I’m using Subversion and not CVS, this has not been too big a task. Just have to move each file then update all the references to it. One by one.

MV3D took a pretty big step forward last night. I ended up working on the problem with player movement and multiple servers. That means I setup two seperate MV3D servers that would share the simulation of the game world between them. In this case, I was simulating a simulation server which would live in the middle part of the network and a player server to which players could directly log in. (Standard server setup for those not in the know– a front end server interfaces with the public (through a firewall). The front end server talks through another firewall to a mid level server (sometimes called application server). The front level server is more or less a proxy whereas the mid level server does a lot of the calculations. The mid level server(s) talk to back end servers (database servers and such) through a third firewall (if you believe Cisco– and keep in mind they’d like to sell you all three firewalls for about the cost of a nice car).

Aaaaanyway, I was using a front end player server and mid level simulation/realm server. So the player connects to the player server which then gets info from the simulation server about where objects are and what they’re doing. There were quite a few bugs.. In things I had written unit tests for even. But it works now with one sort of annoying bug.

It was a real testament to Twisted (and me of course! hah) that connecting through the player server to the sim server (running on a seperate machine) didn’t incur any real noticeable lag. Granted, the two servers were on the same network, but that would normally be the case in a real production level MV3D environment. (in fact, they’d be on a gigabit+ network, none of this silly 100mbit stuff that I have)

Either way, it was really cool (even though it did exactly the same thing that running one server does) because it shows that the multiple server design of MV3D is promising. (and at least mostly functional)

The player server didn’t even have the username and password for the player I logged in as. It got that off the other server as well (and still no noticeable lag).

Any alpha testers? I’ve decided that I don’t want to open up the test to anyone that I don’t know. I should be able to finish the remaining items listed as enhancements for the alpha this weekend.

We’re going to Alpha at least. According to my spiffy new system, I have 2 enhancements and 2 defects to complete before I can officially say the alpha test is ready. Namely: add god-mode world editing, finish game state persisting to database, fix problem where colliding with other players causes viewpoint instability, and fix player movement in a multiple server environment. The last one will be fun to test because I’ll be hooking up a client for the first time into a MV3D game that is running on more than one server. In fact, there hasn’t been a game running on multiple servers yet. The functionality has been tested in Unit Tests and works fine, but I haven’t run two servers for long periods of time like that.

Speaking of “spiffy new system,” you can check it out at:
http://dev.mv3d.com:8080/cgi-bin/trac.cgi/wiki

I got all of character generation (for the alpha test) done last night. That and persisting are the only major hurdles I believe. God mode editing is planned out and ready for programming.

One of the next big hurdles coming up after the brunt of the bugs found in the alpha test are squashed is terrains. It’s hard to have a MMORPG without ground. Since Ogre doesn’t support terrain in any sort of dynamic way, it’s going to be up to me. I am going to try to generate meshes on the fly for patches of terrain which will correspond directly to some objects on the server side. In fact, some of the mesh generation algorithms can be shared between the two since meshes will be required within ODE for collision. ODE does have a terrain collision model, but I think it’s still in the experimental stage and is not part of the main ODE distribution. Eventually, we will switch to that. Anyway, the size and detail of these terrain pieces is going to be up for debate (um.. I’ll be debating with myself) since how well they perform will depend on getting the right size/detail. For those in the know, I should answer the question as to why I’m not going with some sort of progressive mesh for these like CLOD or whatever. The reason is that I’ve seen a few places that are saying that with modern graphics cards and a modern 3d engine, progressive meshes actually are slower than just bruteforce rendering of the terrain. We’ll put that theory to the test. And I fully intend to do a significant terrain load test on both the server and the client. I know ODE can handle it– it did just fine with fairly large terrains in Siegium. Can Ogre handle it? Dunno. (and can Python generate these meshes fast enough?)

That’s probably all for now.

So after some frustrations with getting the physics engine to cooperate, there is now some semblance of normal player movement. You can walk, and turn and when you bump into things, you push them instead of falling over. In fact, you don’t fall over (which is good, and making it work like that was the source of a lot of frustration). The client does get a little wonky when you collide into another player at high speed (need to limit the walking speed still), but if you disconnect and reconnect, it’s normal again.

At this point, you can run the client (still only on linux) then enter your host name, login, and password. Click connect and (if you didn’t mess up the login info) it’ll let you select a character to connect to (since one account can have multiple characters). The server creates any number of characters (I have it set to 10 for now) and it creates a single account that can connect to all of them. Select a character and click “Connect” and you will start playing as that character. You get to see out of the character’s eyes and steer it around with the arrow keys.

The server running on NightEyes was up for 4 days before I took it down. And I only took it down to upgrade it to the newest version. Connecting to it (rather than a server hosted on my desktop machine) doesn’t give any noticeable amount of latency either! So this is good. The question is just what’s the latency connecting to it over the internet? It may be very laggy since I think it updates the position of every moving object 10x per second. In any case, the new version of the server isn’t taking any more CPU time than the old one, so that’s good.

I don’t believe I have that many more things that I need to do for the Alpha Test to start.According to the list, I think I just need to add Chat, In Game Editing, make sure that saving the game world to DB really works, and then setup the Asset downloading mechanism. Oh yeah, and the most important but probably most painful– get it working on Windows. Anyway, I think this means I’m ready to ask the question:

Who wants to be an Alpha Tester? Really, at this point, you will be a QA person. Your mission will be to break things on the server & client and report bugs. The game is not fun. It’s not even a game at this point. I would love to get up to 5 people to start. It would be nice to have people who run on several OS’s (Windows, Mac, Linux). However, Linux people who have 3D cards and know their way around a command line will be the first I’ll need. I actually suspect that the game will work on Mac out of the box-ish, but whoever volunteers for that will need to compile and install things on their system since I won’t be able to produce binaries. So here’s the requirements:

• Fairly decent (P4) Linux, Windows, or Mac system
• 3D Card bought in the past 5 years
• For Mac and Linux, some programming experience (or at least knowledge of how to compile and install software)
• A lot of patience

And here’s what you’ll have to do:

• List your system specs
• Install the software according to instructions
• Run the Unit Tests for the software on your system and report results
• Run the client and report any bugs
• There will probably be several server load tests where we will need to arrange a time for all testers to connect to the server
• Submit bug reports when you find them
• Help narrowing down bugs to determine what the smallest set of actions is to cause them

Any volunteers? I’m only looking for people who are serious about testing. I suspect the test will start officially in a few weeks, but Linux (and maybe Mac) people can probably get in on it earlier.

I may need to turn people down as testers if I get too many (or too many on one OS or whatever). I’ll also be looking at people I know first. Don’t be disappointed. As things progress, I’ll let more and more testers in. Also please forgive me if it’s more than two weeks until the test starts, or if you have an insanely hard time getting the software to work on your system. This is a very early stage test, so there’s going to be lots of bugs and strange problems.

I almost forgot my screenshot!