MV3D Development Blog

November 9, 2007

Get some candy in your eye.

Filed under: Uncategorized — SirGolan @ 5:53 pm

I took a break from bodies to add some eye candy. Bodies are 99% working. They seem to have issues when they are in an octree node that gets split though. Have to check that out. Also have to make a method to limit how fast they update based on something or other.

First up, a lot of objects that use bodies. 2000 to be exact.
Pretty reflections at night Trees, grass, and water. Not too bad for FPS, but not very good either.
Walking through the forest. Trees and water at sunset
A whole lotta trees Too many?

So… This stuff isn’t too impressive because I basically shoe-horned in this Python-Ogre demo. And for some reason, you get pretty clouds in the demo, but I don’t get them. Anyway, this isn’t done in a client server way at all. It’s just attached to the client side of the terrains. Making it into something that comes from the server probably won’t be incredibly hard. The trees will also need integration with physics since you can walk through them right now. There are other optimizations I can do I’m sure to get things going faster– especially tweaking the settings of the tree plugin.

Right now, I need to get back to the server side of things and make bodies work better. Something I really need is an easy way to write unit tests that involve a client and server. I spent a lot of time going back and forth with Bodies making them work when transferred from a server. I more or less had this working before, but it was pretty kludgey (Ogre would get initialized and everything). It would have been nice to be able to write unit tests for the issues that came up.

Anyway, better get back to doing some non eye-candy stuff.

November 8, 2007

Preview

Filed under: Uncategorized — SirGolan @ 6:52 pm

So, I almost have my example Box object that uses bodies going. It works ok, but it sends updates way too often, client side Physics may be not working (hard to tell since it sends updates so often :) ), and you don’t see it in its initial position until it moves. But I took a short break to give you the following pretty picture:

I should have some trees and grass to go along with that soon.

Also, I forget if I mentioned, but I saw that the Python-Ogre guys made wrappers for libnoise. This is very cool because even though MV3D already has a pretty good noise generator, this one is in C++ and therefore will be quite a bit faster. I suspect the new sky plugin uses it for real time clouds (which you can’t see in the screenshot because they are broken for some reason).

November 6, 2007

Linux? No waaay.

Filed under: Uncategorized — SirGolan @ 7:22 pm

Way.

Someone was nice enough to build and package Python-Ogre on Linux. So, now I can run the client there again. Maybe I’ll even package up Python-Ogre + MV3D on Linux if enough people would be interested in such a thing. However, MV3D’s svn trunk is a bit messed right now. Combinator on Windows lied to me when it said I was working in a branch (seriously, it seemed to work and even created MV3D/branches/branchname and put files there, but their .svn/entries points to trunk :(, and I only found out after totally screwing with everything). Yes, I could revert it and try again, but I’ve almost got it fixed.

Anyway, the whole point of this post is the screenshot:

Otherwise, I’m continuing to fix the stuff I broke. That mostly means ripping out my hacky cooperator-ish code and replacing it with real cooperators. Unfortunately, the old code causes Python to crash on Windows, so it has been a little hard to debug. And very odd since MV3D’s server never crashes. It’s a stack overflow that is causing the issue, so increasing the stack size a bit (like on Linux) should get me a maximum recursion depth reached error in Python rather than a crash. But anyway, I know why it’s caused, it’s just finding all the places that cause it and fixing them.

Edit: Trunk fixed… or at least I can log in and walk around now. There may be other issues hiding out.

November 5, 2007

Hello Axiom World

Filed under: Uncategorized — SirGolan @ 12:34 am

So first, I took a look at some more Nevow stuff, and it seems like it should be pretty easy to work with. I didn’t expect otherwise, but you never know.

I just finished writing up a basic data model for the photography website. It was pretty easy. In order to test it, I created some unit tests that store and retrieve things. I figured out how to do various types of relations such as one to many, and many to many. All in all, it’s pretty straight forward and should do the job very well. The next step here is to integrate it with Nevow and then make an admin interface to create data.

The big question is: can I use it on MV3D? I think for 90% of the data, it would be fine. The other 10% is the stuff that changes 20x per second like object position, rotation, and velocities. I have no idea how I’d store that stuff in Axiom as I wouldn’t want to save it to disk so often. Once a minute is probably fine. Maybe there is some sort of parameter I can set for that. I’m definitely open to any ideas here.

As far as other MV3D stuff goes, the new version of Python-Ogre includes a bunch of kick-ass addons such as a really nice dynamic sky, and paged geometry. The paged geometry is what will allow me to have forests with tens of thousands of trees in them (no really, this can display hundreds of thousands of trees on screen without breaking a sweat). There’s also now a kick-ass water texture/shader in the Python-Ogre demos that I can use. Now if I only wrote a nice tree generator that could be used with the paged geometry.

For things that I’ve done, well, I’m creating a base object that uses bodies. It is < 200 lines of code, which is a good sign. I also refactored out a lot of stuff related to object views and figured out how that whole mess will work with bodies and visual objects. Basically, items (objects, areas, realms, etc) in MV3D can be Viewable. Something that is viewable has a sister view class that contains only the attributes that are needed for the client. The reasoning for this is that you don't want the client knowing everything the server knows about an object. That leads to cheating. Right now, viewable objects that don't use bodies update position and rotation of their views all the time. This isn't slow, but it could become so. Instead, with bodies, I've decided that the body level and below is all public data that the client can know about. So, an object and its view share a reference to a body (or more than one). The body can have visual objects attached to it, which become models and such on the client. On the client side, the body updates the position/rotation of its visual objects as needed based on client side physics and updates from the server. It seems to make a lot of sense to me, but we'll see how it works soon enough.

On the subject of portals, it actually looks like I can do some render to texture work in order to support smaller portals visually. Basically, I should be able to create a flat object and render whatever is behind the portal to it. It is theoretically possible to only render to it when it is visible to save some cpu cycles.

The next step here of course is getting a working object using bodies on the server and client. But first, housework.

Edit: Persistence Thoughts
Right now, bodies don’t actually have a member variable called “position” or anything. They just get/set that information directly in the physics engine. Maybe I could define position/rotation/velocities as Axiom attributes and then once a minute start up a coiterator that goes through all the items on the server and calls a method on them that would cause the bodies to do something like self.position = self.odeBody.getPosition(). Does that seem like a reasonable solution? I don’t know Axiom nearly well enough to say if that would be fast and not dumb.

November 2, 2007

So simple, it must be right!

Filed under: Uncategorized — SirGolan @ 2:03 pm

I just figured out how to handle collisions between areas with separate coordinate systems. Give gateways their own physics space. Whenever a body gets near the gateway, it adds colliders in that space. After checking collisions in the area, all the bodies in the gateway space are translated through the gateway, and then the gateway space is checked for collisions against the gateway’s destination area. Then just translate the bodies back out of the gateway afterward. They’ll have all the correct collision info generated, and it should be a pretty quick collision detection. Changing the positions and back is a little annoying, but I really think this is the absolute best way to handle it.

November 1, 2007

Hello Nevow Web

Filed under: Uncategorized — SirGolan @ 7:53 pm

So, it took under 30 minutes for me to follow the tutorial for hello world here. Would have been much faster, but that is a fairly outdated tutorial. Mostly the issue is that twistd has changed a lot. Here’s what I ended up with for the twistd plugin code if anyone is interested:


from zope.interface import implements

from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker
from twisted.application import internet
from twisted.python.usage import Options

from nevow import appserver

import helloworld

class HelloOptions(Options):
    """
    Command-line options for the web server.
    """

    optParameters = [
        ('config-file', 'c', None, 'The file to load configuration from.'), ]

class HelloWorldPlugin(object):
    """
    The plugin that starts the website
    """
    implements(IServiceMaker, IPlugin)
    tapname = "helloworld"
    description = "hello website"
    options = HelloOptions

    def makeService(self, options):
        self.site = appserver.NevowSite(helloworld.HelloWorld())
        self.webServer = internet.TCPServer(8080, self.site)

        return self.webServer

thePlugin = HelloWorldPlugin()

With that finished, it’s off to the ‘hello Axiom-world’. But first, some MV3D stuff.

I’ve been doing a lot of refactoring, reformatting, and re-testing of code. I started changing things up so that instead of inheriting from something like odeObject in order to be an object in a 3D environment that has a physical representation, I’m adding Body classes that handle physical and visual representations. This way, those can just get sent to the client as they will have no data in them that the client should not see.

Right now, there is a FastObject3D class that you can inherit from that basically aggregates its position updates and queues them for all objects going to a connection. The queue will drop updates if there are too many, and updates sent to the client are thinned down by a few factors such as distance between the PC and the object that is moving. This was a direct result of the load testing I did a while back with thousands of objects. The load ended up not being the physics engine once I got things straightened out there, but instead, it was in serializing all the updates to send them down the wire. I’m thinking that I’ll move FastObject3D’s logic to the Body classes more or less, and then add some extras to it. One thing I’d like to do to make user experience better is whenever there is a position/rotation update that is significantly different from what the position/rotation is on the client, the client will average that in over a few frames to make it less jerky. I’ve put in some thinking on a related subject of “the client is usually right about where he’s looking,” but I’m still not ready to make a decision on that one yet. The other thing I may add in if I get really fancy is to have the update queue send info to clients/other servers over UDP as just a stream of bytes. That could speed it up even more. Though right now, I don’t really have speed or bandwidth problems, so that would be a premature optimization, imho.

One more thing to mention about the Body classes. They need to handle collisions between objects in adjacent areas that have different coordinate systems. To explain, MV3D has areas, which don’t have a size or a position (except OctreeArea since Octrees require it). You can connect areas with gateways, and one thing you can do with gateways is specify a translation or rotation for any object that passes through the gateway. This is absolutely essential since MV3D uses floats instead of fixed point. Without it, as you got further away from (0,0,0), the accuracy of the physics will get worse because of how floating point numbers work. Anyway, now there is no need to extend things very far from (0,0,0). Just make a gateway at (0,0,10000) that goes to (0,0,0) in its destination. Anyway, this brings about an interesting challenge when an object is on the border between two areas that are in different coordinate systems. The physics engine only understands one coordinate system. Currently, only Bipeds can handle collisions between areas in different coordinate systems. They do it by creating duplicate physics bodies in each area near their position. This works, but is not very efficient and also causes some strange artifacts when passing between areas. It’s not efficient because the “main” body has to update the sub bodies positions every frame. I’ve tried just adding colliders in the other areas, but that doesn’t hold up if the object rotates (because they act like two objects connected by a very long pole). The same thing if I create two bodies and use a joint to keep them in sync. Basically, I’m not sure how best to handle this, and it is fairly important. I need to figure this out before I can move on basically, but at least with Bodies, it’ll be all kept in the same place.

What’s next after I get all of that working? I’ll have to yank out 90% of all the currently created objects’ code and replace it with Bodies. That will be fun. Then, well I think I want to create a new object using Bodies to see if it is sufficiently painless. I’m thinking of making a sword object that is modular and is derived from three models so that I can make a random sword generator (after making multiple blades, hilts, and guards). From there, I’d like to do some more load testing, some refining of the world editing tools, and lots of other things. I really need to get v0.4 out the door though, so I’ll have to draw the line somewhere.

All these changes have been taking a while to make because other than not having much time to work on them, I’m also reformatting modules that use the old formatting style, switching all unit tests to trial, adding unit test coverage where there is none, and refactoring quite a bit. More later.

October 30, 2007

A website based on Twisted, Nevow, and Axiom…

Filed under: Uncategorized — SirGolan @ 7:21 pm

I’m about to embark on a not too interesting project (compared to MV3D) that involves creating a website for my sister’s fiancee’s photo business ( http://www.coreyengferphotography.com ). For various reasons, this is a great time for me to learn Nevow and Axiom. Since my blog is generally only about MV3D, I figured I’d shake things up a bit and share my experiences and also see just how quickly I can build the site. Probably pretty boring for most people, so consider this a warning. :)

For the record, Twisted is an event driven networking engine for Python (quoting from their website). Mostly, I’ll be using twisted.web (or web2). Nevow is a Python based framework to build web applications, more or less. Finally, Axiom is an object based data store a little like the one I wrote.

I’ve never used Nevow or Axoim before and all I know about Nevow is that Glyph says it could replace the HTML library I wrote. I’ve actually written a template engine in python. Corey’s site uses said templating engine. It’s kinda clunky and tries to be both a template engine and a CMS and is not wonderful at either. I’ve looked at Axoim for MV3D and want to give it a try to see how it works.

The website is fairly simple. The only real moving parts being a shopping cart that interfaces with PayPal, the ability to list photos and for users to customize how they want them (8×10 print vs holiday card print, etc).

There’s one more thing to add, which is a front page flash gallery that cycles through some random photos. I’ve managed to completely avoid writing any flash apps so far, but it looks like I can’t avoid it any more.

I’d like to give Corey the ability to manage the content on the site so that he doesn’t have to come to me to change his about page or whatnot. About the only other thing I can think of is linking some (all?) photos to Corey’s Etsy store.

So, with that said, part one involved setting up apache for reverse proxying. Since I’ll be using Twisted’s stand-alone webserver and I only have one IP address from my ISP, I’ve had to create a virtual host in Apache that basically remaps http://www.coreyengferphotography.com to http://localhost:8080 (or whatever port I run the server on). I recently did this for trac.mv3d.com, so it’ll just be a matter of doing this:

<VirtualHost *>
ServerName www.coreyengferphotography.com
ProxyPass / http://localhost:8080
ProxyPassReverse / http://localhost:8080
</VirtualHost>

October 12, 2007

On second thought..

Filed under: Uncategorized — SirGolan @ 8:26 pm

It should have occurred to me that releasing a tool to beautify code was a bad idea if the tool’s code wasn’t very beautiful. Anyway, Stylin was just supposed to be an afternoon project so I could use it to reformat MV3D code. I’ve gone and refactored a bit and put unit tests in the zip file. It’s still got the same bugs it seems (which probably means I’m not testing enough strange cases). Mostly, there first time it goes to add extra spaces, it adds a bit too many. It’s in the same location as before. Like I said, it could use even more cleaning up.

Also, convinced me to have another look at Axiom for storage. He’s apparently already built something like my DataStore and ran into problems with it. The usual sequence is: him telling me to do something, me ignoring it, and then 2 or so years later realizing he was right all along. This time, I’ve decided to cut out the 2 or so years and at least take another look at it and try using it in MV3D.

October 7, 2007

Save that thought..

Filed under: Uncategorized — SirGolan @ 9:28 pm

I’d like to take a moment to talk about persistence. In general, what I mean by that is storing stuff to disk or some other medium that persists (hah) beyond when the application is shut down. Back in July, I mentioned a little about the issues I was having. Mostly, storing to database isn’t fast enough for me because of the amount of data and how it needs to be stored. I looked at a lot of options including pretty much all of the Python ORMs. I did however miss Zope’s object store, but… How better to learn about Object Oriented Data Stores than to write one myself?

I wrote the actual OODS in about a weekend. Currently, it generates a unique id for an instance of a class and from that a filename to pickle that class to. That sounds pretty easy, right? Not so much. It has to handle nested classes (and save the inner classes to their own files). It has to handle circular references within classes, and it has to be pretty solid. What I came up with does all this.

In addition, I thought it would be nice to include some database-like features such as performing queries. So, you can create indexes on class member variables and perform queries on them as you like. The query language is pretty simple and straight forward, I think:


ds = DataStore()
result = ds.query("Users").fetchResults(q.firstname == "Frank" & q.age >= 20)

That would return all users with “Frank” as their first name who were also over 20 years old. Fun stuff. Queries run off of indexes, so they should be pretty darn fast. Speaking of fast, I’d like to mention that for a fair sized world, saving only takes a couple of seconds. It used to be a couple of minutes. Plus, I haven’t noticed any jittering due to the physics engine on the server not being able to keep up as was normal when saving to DB.

There are a lot of future features I’d like to add. One of the features that has been designed in but not implemented is the ability for it to connect to a store over PB, and automatically do so when trying to open a store that is locked. A little bit simpler than that is to make a configurable storage tree so you can specify different data stores depending on class or other values. In particular, this would be good for sectioning off things that actually do fit into a database to a real database. You could also use it for high availability and other fun things that MV3D doesn’t need in its persistence mechanism. Remember (speaking to myself too here), MV3D only persists stuff a) for simplicity during development, and b) in case master and secondary servers go down.

Something that may make sense to do soon is to send the store index and any query indexes to a database because a database would be very good at handling that. And finally, creating a method to do transactions and either commit them or roll them back would be pretty nifty. Once again, this isn’t something that really helps MV3D’s needs at all. Oh yeah, one more thing, doing the relational db equiv of joins may be handy, dunno.

Anyway, I’m considering releasing this separate from MV3D. I’m interested to hear what people think of that possibility. I suspect it would have uses outside of 3D games, especially if some of the features I mentioned were implemented.

Oh yes, of course all of this is compatible with Twisted and everything either runs async or deferred into a thread.

I’m still having a little trouble realizing that even though I’m not currently running my dev MV3D server, I can start it up at any time and hop right in to the world the same way it was. It’s actually been a timesaver while testing the issues I had integrating the datastore into MV3D.

Speaking of that, I mentioned that I wrote the datestore in about a weekend? Well, it’s taken a lot of work to integrate it into MV3D, but it is working very well now. Most of the problems relate to the unpresistability of ode stuff. What happens now is everything that uses physics registers a callback that gets called when everything is loaded. The callback sets up its physics attributes. There are all sorts of interdependencies to worry about, and I haven’t looked at a lot of them in a year, so it was not fun to fix. For instance, one of the issues I solved today was that after the first save/restore, objects other than people would fall through the ground. I examined the data on the server many times– usually this happens when the objects aren’t in the same physical space as the rest of the world, but this wasn’t the case. It ended up being that ODE areas need to register themselves with their realm so that the realm can handle collision detection inter and intra area. Yeah that took quite a few hours of frustration to finally track down. It all works great now though.

What’s next?

One feature I didn’t mention about the datastore: vacuuming. When things get “deleted” from the datastore, they are removed from the indexes, but not from disk. So every once in a while, you’ll want to check all the files on disk against the index and delete them if they aren’t in there.

After that, I’m thinking about some client enhancements. Mostly stuff to make testing faster such as a config file for the client that lets me skip the Ogre screen size selection and connect/select pc windows. With that will probably come some ui stuff for client configuration as well as having the client handle the list of connections. Currently, it’s hard coded, so if you don’t want one that’s on the list, you need to enter it manually.

I’m also thinking about refactoring. There’s a lot that could use it, and it would be nice to catch up on unit tests to go along with refactored code (which would include switching the unit tests to use trial). I need to find a way to do integration type testing for MV3D.. Anyone have ideas? :) I have a non graphical client that can connect to a server as a pc and do stuff. It’s annoying, but I guess I could go with that approach.

That’s all for now. Still need help in the art department. :-/

October 6, 2007

That Code Be Stylin’!

Filed under: Uncategorized — SirGolan @ 1:36 pm

Soo.. Been a while, eh? Long time no see. Things have been pretty busy with work, but I’ve gotten a little bit of time to work on MV3D stuff. I’m not going to talk about that now. If you’re lucky (or unlucky if you find me boring), I’ll write more later today about MV3D.

Something that really irks me is ugly code. There’s a lot of MV3D code that I consider fugly. Most of the reason is the formatting (darn me and my C++ habits), so rather than go through 30kloc of manual reformatting, I decided that was a job for a program. I asked around a bit and it doesn’t seem like anyone’s written a Python code reformatter (or at least one that can format to Divmod coding style), so I’ve made Stylin.

Now you can have it, and may your code be stylin’ too!

Here you go. (Sorry for the zip file, but my webserver is configured to try and execute .py files)

What does it do?

Let’s start by saying what it doesn’t (yet) do:

  • Wrap lines at 79 chars
  • Properly add blank lines around import statements at the top of the
  • Properly add blank lines around random code that isn’t indented (and isn’t a class/function)
  • Move your indents to the top of the file (this is probably not a good idea anyway)
  • Make your algorithms cleaner
  • Occasionally it has an issue with mismatched quotes within a docstring
  • Unless you do something with mismatched quotes like above, it does not break your code.

    So with that in mind, what does it do?

  • Put spaces around operators (but not in keyword arguments on function defs)
  • Re-indent everything to use spaces and not tabs (can even handle mixes)
  • Add blank lines between classes, methods, etc
  • Reformat doc strings
  • Add doc strings where there are none (”You need a docstring here!” :) )

    Enjoy!

  • « Older PostsNewer Posts »

    Powered by WordPress