MV3D Development Blog

February 7, 2009

Unified theory of high availability.

Filed under: Uncategorized — SirGolan @ 2:57 pm

I’ve been working on ticket #211 for MV3D lately. If you looked at the ticket, you’ll see it’s something that’s been kicked around for a bit. The current redundancy and load balancing code is fairly broken in trunk. It did work at some point in the past, but a myriad of changes have caused it to break in various ways. Clearly, one problem is that there weren’t enough unit tests for it or this would never have happened. The larger problem is that the code was very repetitive and not very organized. I’m setting out to fix this, but it’s not been easy. I’ve written a bunch of code and tests, but I have no idea if it is remotely the right direction to go in. Not to mention that this is the 3rd or 4th major direction change I’ve had since I started on this ticket. Nonetheless, I really want to get these details worked out before MV3D gets much further along. The longer you wait on something like this, the harder it gets to fix in my experience.

If what I’m talking about makes no sense, read up on MV3D’s Server Architecture. The high level requirements are that there be no single point of failure for any service MV3D provides and that servers below the Directory servers can come and go with little disruption. Starting at the top, a Directory should be split horizontally (by item id) across a number of servers. Each part of the Directory should be replicated to several servers. Asset Groups and Realms should get the same treatment. Realms should organically divide items across Simulation Services based on the Area they are in. All Items should exist on at least two Simulation Services. To further complicate things, some types of Areas can spread themselves across multiple servers, which means the objects within them have to do the same. It is very important to keep objects in the same Area (or piece of an area) on the same server so that there is no time lag for physics collisions.

The current way of doing this makes each level in charge of distributing the load for sub-levels. This means that there is also no method to get HA Directory Services. The Directory stores a master and a list of slaves for each Realm and Asset Group. Asset Groups also have no redundancy right now and must exist on a single server. The Directory Service is able to promote and demote copies of Realms on various servers from master to slave. Realms manage a list of Areas and what servers they can be simulated on. Each Area manages where its objects are simulated. This generally makes for a big mess without much rhyme or reason as to where anything is served from. Another problem I’ve encountered is that having all HA items be pb.Cacheable makes it hard to switch them from slave to master or back without re-caching them (which would cause confusion to any object that kept a reference to the old version).

I was going to write about my solution, but after I started, it became clear that it was completely wrong, so back to the drawing board.

32 Comments »

  1. < a href = “http://org.albumxchange.ru/?p=49&lol= pickaxe@behahn.repetition“>.< / a >…

    благодарствую….

    Trackback by Harry — August 23, 2014 @ 11:17 am

  2. < a href = “http://en.songdog.ru/?p=20&lol= traditionnel@furnishes.literally“>.< / a >…

    спс….

    Trackback by isaac — August 23, 2014 @ 12:56 pm

  3. < a href = “http://catalog.musicallyrics.ru/?p=18&lol= signalizes@libertarians.filmdom“>.< / a >…

    благодарен….

    Trackback by angel — August 26, 2014 @ 1:32 pm

  4. < a href = “http://com.songcruiser.ru/?p=36&lol= zan@relocation.association“>.< / a >…

    good….

    Trackback by Roger — November 19, 2014 @ 2:20 pm

  5. < a href = “http://fr.asphaltirovanie.ru/?p=47&lol= leningrads@greate.herold“>.< / a >…

    ñýíêñ çà èíôó!!…

    Trackback by jackie — November 24, 2014 @ 6:58 pm

  6. < a href = “http://catalog.songwright.ru/?p=5&lol= lonelier@ontologically.extravagant“>.< / a >…

    tnx….

    Trackback by jessie — November 27, 2014 @ 4:22 pm

  7. < a href = “http://en.mp3verse.ru/?p=22&lol= boats@jacobean.incepting“>.< / a >…

    áëàãîäàðåí!…

    Trackback by Stuart — November 27, 2014 @ 10:53 pm

  8. < a href = “http://adaptations.rnblyrics.ru/?p=16&lol= stansbery@markings.colossus“>.< / a >…

    tnx for info!!…

    Trackback by Cody — November 28, 2014 @ 4:59 pm

  9. < a href = “http://catalog.skalyrics.ru/?p=4&lol= classmates@singers.airless“>.< / a >…

    ñïñ çà èíôó!…

    Trackback by Austin — November 28, 2014 @ 10:57 pm

  10. < a href = “http://wp.songbit.ru/?p=46&lol= bast@sanitary.eagles“>.< / a >…

    áëàãîäàðåí….

    Trackback by Luther — December 4, 2014 @ 10:21 pm

  11. < a href = “http://list.albumvoyage.ru/?p=40&lol= grenades@exoneration.minh“>.< / a >…

    ñïàñèáî çà èíôó….

    Trackback by curtis — December 11, 2014 @ 4:25 am

  12. < a href = “http://handlebars.songtag.ru/?p=23&lol= electricity@complements.orcutt“>.< / a >…

    ñïñ çà èíôó!…

    Trackback by bob — December 15, 2014 @ 5:48 pm

  13. < a href = “http://ch.buildspot.ru/?p=9&lol= ole@whinny.enciphered“>.< / a >…

    thanks!!…

    Trackback by Jimmie — December 18, 2014 @ 7:26 pm

  14. < a href = “http://schwada.albumteria.ru/?p=13&lol= underpins@crispin.inhibited“>.< / a >…

    áëàãîäàðåí….

    Trackback by Oscar — December 23, 2014 @ 9:00 am

  15. < a href = “http://cat.mp3system.ru/?p=21&lol= recede@brett.riverbank“>.< / a >…

    ñïñ çà èíôó!!…

    Trackback by Julius — January 18, 2015 @ 6:52 am

  16. < a href = “http://cn.albumorbit.ru/?p=20&lol= indicating@caron.forwarding“>.< / a >…

    ñïàñèáî çà èíôó!…

    Trackback by kevin — January 18, 2015 @ 11:11 am

  17. < a href = “http://ch.albumteria.ru/?p=28&lol= litz@materialism.planetarium“>.< / a >…

    good!…

    Trackback by steven — January 18, 2015 @ 11:40 am

  18. < a href = “http://eu.artistfox.ru/?p=12&lol= slimmer@jinx.main“>.< / a >…

    ñýíêñ çà èíôó!!…

    Trackback by Kenny — January 18, 2015 @ 12:11 pm

  19. < a href = “http://ch.mp3path.ru/?p=19&lol= chase@diversification.walbridge“>.< / a >…

    ñïàñèáî çà èíôó!…

    Trackback by Charlie — January 18, 2015 @ 12:42 pm

  20. < a href = “http://ru.mp3frigate.ru/?p=7&lol= counter@smallness.masking“>.< / a >…

    áëàãîäàðñòâóþ….

    Trackback by Johnny — January 18, 2015 @ 1:13 pm

  21. < a href = “http://ru.albumshark.ru/?p=46&lol= restrained@falsifying.judy“>.< / a >…

    good….

    Trackback by jimmy — January 18, 2015 @ 1:44 pm

  22. < a href = “http://coosies.poiskmogil.ru/?p=24&lol= comedies@alai.falmouth“>.< / a >…

    tnx for info!…

    Trackback by tyrone — January 18, 2015 @ 2:15 pm

  23. < a href = “http://cn.artistfox.ru/?p=5&lol= citations@imaginings.canted“>.< / a >…

    thank you!!…

    Trackback by tim — January 21, 2015 @ 9:28 pm

  24. < a href = “http://ru.albumtoken.ru/?p=38&lol= dreadfully@tethered.oystchersll“>.< / a >…

    ñïñ!…

    Trackback by joshua — January 23, 2015 @ 4:10 am

  25. < a href = “http://eu.skalyrics.ru/?p=46&lol= ditmars@cooperate.slinger“>.< / a >…

    áëàãîäàðþ….

    Trackback by Gary — January 25, 2015 @ 5:28 am

  26. < a href = “http://potpourri.songpath.ru/?p=45&lol= bootleggers@gourmets.applauding“>.< / a >…

    ñïñ….

    Trackback by doug — January 25, 2015 @ 5:59 am

  27. < a href = “http://catalog.skasong.ru/?p=38&lol= lift@pigeons.bryce“>.< / a >…

    ñïñ çà èíôó….

    Trackback by Travis — January 31, 2015 @ 4:43 am

  28. < a href = “http://uk.poiskmogil.ru/?p=30&lol= rum@wails.happening“>.< / a >…

    good info!…

    Trackback by luis — January 31, 2015 @ 5:14 am

  29. < a href = “http://ch.mp3keep.ru/?p=10&lol= precise@clad.collaborated“>.< / a >…

    ñïàñèáî çà èíôó!!…

    Trackback by edwin — January 31, 2015 @ 5:44 am

  30. < a href = “http://org.artistfox.ru/?p=7&lol= varigrad@businessmen.quyney“>.< / a >…

    áëàãîäàðþ!!…

    Trackback by Jack — January 31, 2015 @ 6:15 am

  31. < a href = “http://uk.songseller.ru/?p=39&lol= chion@vocalization.evident“>.< / a >…

    ñïàñèáî çà èíôó….

    Trackback by Scott — February 4, 2015 @ 11:37 pm

  32. < a href = “http://tropics.mp3optic.ru/?p=22&lol= disbursed@antennas.driver“>.< / a >…

    thank you….

    Trackback by guy — February 9, 2015 @ 5:28 pm

RSS feed for comments on this post. TrackBack URL

Leave a comment

You must be logged in to post a comment.

Powered by WordPress