MV3D Development Blog

I’m about to embark on a not too interesting project (compared to MV3D) that involves creating a website for my sister’s fiancee’s photo business ( http://www.coreyengferphotography.com ). For various reasons, this is a great time for me to learn Nevow and Axiom. Since my blog is generally only about MV3D, I figured I’d shake things up a bit and share my experiences and also see just how quickly I can build the site. Probably pretty boring for most people, so consider this a warning.

For the record, Twisted is an event driven networking engine for Python (quoting from their website). Mostly, I’ll be using twisted.web (or web2). Nevow is a Python based framework to build web applications, more or less. Finally, Axiom is an object based data store a little like the one I wrote.

I’ve never used Nevow or Axoim before and all I know about Nevow is that Glyph says it could replace the HTML library I wrote. I’ve actually written a template engine in python. Corey’s site uses said templating engine. It’s kinda clunky and tries to be both a template engine and a CMS and is not wonderful at either. I’ve looked at Axoim for MV3D and want to give it a try to see how it works.

The website is fairly simple. The only real moving parts being a shopping cart that interfaces with PayPal, the ability to list photos and for users to customize how they want them (8×10 print vs holiday card print, etc).

There’s one more thing to add, which is a front page flash gallery that cycles through some random photos. I’ve managed to completely avoid writing any flash apps so far, but it looks like I can’t avoid it any more.

I’d like to give Corey the ability to manage the content on the site so that he doesn’t have to come to me to change his about page or whatnot. About the only other thing I can think of is linking some (all?) photos to Corey’s Etsy store.

So, with that said, part one involved setting up apache for reverse proxying. Since I’ll be using Twisted’s stand-alone webserver and I only have one IP address from my ISP, I’ve had to create a virtual host in Apache that basically remaps http://www.coreyengferphotography.com to http://localhost:8080 (or whatever port I run the server on). I recently did this for trac.mv3d.com, so it’ll just be a matter of doing this:

<VirtualHost *>

  ServerName www.coreyengferphotography.com

  ProxyPass / http://localhost:8080

  ProxyPassReverse / http://localhost:8080

</VirtualHost>


	

It should have occurred to me that releasing a tool to beautify code was a bad idea if the tool’s code wasn’t very beautiful. Anyway, Stylin was just supposed to be an afternoon project so I could use it to reformat MV3D code. I’ve gone and refactored a bit and put unit tests in the zip file. It’s still got the same bugs it seems (which probably means I’m not testing enough strange cases). Mostly, there first time it goes to add extra spaces, it adds a bit too many. It’s in the same location as before. Like I said, it could use even more cleaning up.

Also, convinced me to have another look at Axiom for storage. He’s apparently already built something like my DataStore and ran into problems with it. The usual sequence is: him telling me to do something, me ignoring it, and then 2 or so years later realizing he was right all along. This time, I’ve decided to cut out the 2 or so years and at least take another look at it and try using it in MV3D.

I’d like to take a moment to talk about persistence. In general, what I mean by that is storing stuff to disk or some other medium that persists (hah) beyond when the application is shut down. Back in July, I mentioned a little about the issues I was having. Mostly, storing to database isn’t fast enough for me because of the amount of data and how it needs to be stored. I looked at a lot of options including pretty much all of the Python ORMs. I did however miss Zope’s object store, but… How better to learn about Object Oriented Data Stores than to write one myself?

I wrote the actual OODS in about a weekend. Currently, it generates a unique id for an instance of a class and from that a filename to pickle that class to. That sounds pretty easy, right? Not so much. It has to handle nested classes (and save the inner classes to their own files). It has to handle circular references within classes, and it has to be pretty solid. What I came up with does all this.

In addition, I thought it would be nice to include some database-like features such as performing queries. So, you can create indexes on class member variables and perform queries on them as you like. The query language is pretty simple and straight forward, I think:


ds = DataStore()
result = ds.query("Users").fetchResults(q.firstname == "Frank" & q.age >= 20)


That would return all users with “Frank” as their first name who were also over 20 years old. Fun stuff. Queries run off of indexes, so they should be pretty darn fast. Speaking of fast, I’d like to mention that for a fair sized world, saving only takes a couple of seconds. It used to be a couple of minutes. Plus, I haven’t noticed any jittering due to the physics engine on the server not being able to keep up as was normal when saving to DB.

There are a lot of future features I’d like to add. One of the features that has been designed in but not implemented is the ability for it to connect to a store over PB, and automatically do so when trying to open a store that is locked. A little bit simpler than that is to make a configurable storage tree so you can specify different data stores depending on class or other values. In particular, this would be good for sectioning off things that actually do fit into a database to a real database. You could also use it for high availability and other fun things that MV3D doesn’t need in its persistence mechanism. Remember (speaking to myself too here), MV3D only persists stuff a) for simplicity during development, and b) in case master and secondary servers go down.

Something that may make sense to do soon is to send the store index and any query indexes to a database because a database would be very good at handling that. And finally, creating a method to do transactions and either commit them or roll them back would be pretty nifty. Once again, this isn’t something that really helps MV3D’s needs at all. Oh yeah, one more thing, doing the relational db equiv of joins may be handy, dunno.

Anyway, I’m considering releasing this separate from MV3D. I’m interested to hear what people think of that possibility. I suspect it would have uses outside of 3D games, especially if some of the features I mentioned were implemented.

Oh yes, of course all of this is compatible with Twisted and everything either runs async or deferred into a thread.

I’m still having a little trouble realizing that even though I’m not currently running my dev MV3D server, I can start it up at any time and hop right in to the world the same way it was. It’s actually been a timesaver while testing the issues I had integrating the datastore into MV3D.

Speaking of that, I mentioned that I wrote the datestore in about a weekend? Well, it’s taken a lot of work to integrate it into MV3D, but it is working very well now. Most of the problems relate to the unpresistability of ode stuff. What happens now is everything that uses physics registers a callback that gets called when everything is loaded. The callback sets up its physics attributes. There are all sorts of interdependencies to worry about, and I haven’t looked at a lot of them in a year, so it was not fun to fix. For instance, one of the issues I solved today was that after the first save/restore, objects other than people would fall through the ground. I examined the data on the server many times– usually this happens when the objects aren’t in the same physical space as the rest of the world, but this wasn’t the case. It ended up being that ODE areas need to register themselves with their realm so that the realm can handle collision detection inter and intra area. Yeah that took quite a few hours of frustration to finally track down. It all works great now though.

What’s next?

One feature I didn’t mention about the datastore: vacuuming. When things get “deleted” from the datastore, they are removed from the indexes, but not from disk. So every once in a while, you’ll want to check all the files on disk against the index and delete them if they aren’t in there.

After that, I’m thinking about some client enhancements. Mostly stuff to make testing faster such as a config file for the client that lets me skip the Ogre screen size selection and connect/select pc windows. With that will probably come some ui stuff for client configuration as well as having the client handle the list of connections. Currently, it’s hard coded, so if you don’t want one that’s on the list, you need to enter it manually.

I’m also thinking about refactoring. There’s a lot that could use it, and it would be nice to catch up on unit tests to go along with refactored code (which would include switching the unit tests to use trial). I need to find a way to do integration type testing for MV3D.. Anyone have ideas? I have a non graphical client that can connect to a server as a pc and do stuff. It’s annoying, but I guess I could go with that approach.

That’s all for now. Still need help in the art department. :-/

Soo.. Been a while, eh? Long time no see. Things have been pretty busy with work, but I’ve gotten a little bit of time to work on MV3D stuff. I’m not going to talk about that now. If you’re lucky (or unlucky if you find me boring), I’ll write more later today about MV3D.

Something that really irks me is ugly code. There’s a lot of MV3D code that I consider fugly. Most of the reason is the formatting (darn me and my C++ habits), so rather than go through 30kloc of manual reformatting, I decided that was a job for a program. I asked around a bit and it doesn’t seem like anyone’s written a Python code reformatter (or at least one that can format to Divmod coding style), so I’ve made Stylin.

Now you can have it, and may your code be stylin’ too!

Here you go. (Sorry for the zip file, but my webserver is configured to try and execute .py files)

What does it do?

Let’s start by saying what it doesn’t (yet) do:

Add doc strings where there are none (”You need a docstring here!” )

Enjoy!