The technology of Guild Wars

Being an analyst has its perks, the main one being that you get to have some really interesting conversations. And so I recently had the chance to interview Mike Or rien and Pat Wyatt, two of the founders and lead programmers for ArenaNet, makers of the Guild Wars MMORPG (Massively MultiPlayer Online Role-Playing Game).

If you play games of this sort, its surely obvious to you why you should care. But if you dont, maybe you should be interested anyway. After all, Guild Wars is a graphics-intensive SaaS offering that easily supports 100,000 simultaneous users, while managing a gig or so of fat client even over dial-up speeds. Every user is a potential hacker, whether for fun or actual real-world cash profit, although we didn actually talk about security very much. And ArenaNet provides all this on a relatively shoestring budget; in particular, Guild Wars subscription fees are precisely $0.

Note: If youre familiar with video RPGs but not MMORPGs, please just read on; the main salient difference is that in an MMORPG lots of people play at once. If youre familiar with other MMORPGs but not Guild Wars, theres one GW-specific design feature you should know of ?Guild Wars is instanced. That is, a group of 1-24 players will get their own private copy of the game world. Any changes to the environment (e.g. player movement, monster-killing) are made only by them.

And if youre not familiar with video RPGs at all, I suggest you stop working quite so hard and go play.

Here are some of the technical highlights I managed to glean in a brief half-hour conversation:

The client doesns do much except graphical presentation and input preparation (including encryption). This is a deliberate security choice. They assume the game client is infinitely hackable, and hence limit its power to sending a small number of well-defined messages.

Their servers run on blades ?originally all IBM, now more varied. Generally these are 2.8 gigahertz dual-processor Xeon IBM blades with 2.5 gigs of RAM.

There are a number of different categories of servers. These include: Authorization/log-in, game update download, actual gameplay, database cache, DBMS, several secondary game functions (e.g. guild membership, tournaments), and watchdogs overseeing the rest.

Gameplay servers can support 2500-3500 users each, with the main limitation being addressable memory. (They run on 32-bit Microsoft Windows.) 3500 users is a bit uncomfortable. This matches well against the fact that character records are several 10s of kilobytes, for reasons I explain in this companion post on Guild Wars data management.

There are 4 ?million total lines of code.

Their connectivity design goal is .0045 megabits/sec/user, and theyre close to achieving it. Dial-up is a fairly realistic way to play. (That said, as a player I can tell you it creates some lags that make a dial-up user less than idea to be partnered with.)

Reliability is not just a nice thing for users, but an actual game integrity concern. There are possible exploits in MMORPGs that involve deliberately crashing a server when a goods-trade transaction is partway through. ArenaNet fights this both by focusing on transaction integrity (but not in the way you may think; I refer you again to the database-focused companion post) and by generally trying to make their database cache servers extremely reliable. Database cache servers typically stay up 150-200+ days at a time.

*Note: There is a high level of motivation for cheating in MMORPGs. One of the biggest is that professionals in low-wage but well-wired countries acquire fictional gold, so that they can sell it for real money. Gold sellers are a huge concern to ArenaNet.

The database technology of Guild Wars

I have the enviable task of researching online game and virtual world technology for an upcoming Network World column. My first interview, quite naturally, was with the lead developers of a game I actually play ?Guild Wars. The overview is in another post; that may provide context for this one, which focuses on the database technology. (I also did a short post just on the implications for Guild Wars players.) It also has a brief description of what Guild Wars is ?namely, a MMORPG (Massively MultiPlayer Role-Playing Game) with the unusual feature that most of the game world is instanced rather than utterly shared.

First, some scope. ArenaNet (Guild Wars?developer, now a subsidiary of NCsoft) runs Microsoft SQL Server, mainly Enterprise Edition, having just switched to 2005 4 months ago. They run 1500-2500 transactions/second all day, spiking up to 5000 in their busiest periods. They have no full-time DBA, and when the developers started this project they didn know SQL. They e only had one major SQL Server failure in the 2+ years the game has been running, and that was (like most of their bugs) a network driver problem more than an issue with the core system.

As for whats going on ?there are a few different kinds of database things that happen in an instanced MMORPG.

Game characters acquire loot; i.e., there are inventory transactions. These are commonly one-sided. Goods are created pretty much out of thin air ?or rather thin electricity ?and so there no notion of matching a deletion from stock against consumption in manufacture or anything like that. Similarly, they buy and sell at merchants and traders and so on. One character carries around 60 different items, each of which is a combination of several tokens or attributes chosen from a finite list.
Characters evolve. They gain power. The portion of the game map they can see increases. They experience milestone events in the game. Etc.
There are temporary changes to the game world as characters explore areas. These need to be maintained as state while the character is online and in the specific area; but because the game is instanced, they dons need to be persisted beyond that (with minor exceptions).
There are a variety of secondary systems with ordinary transactions, such as guild rosters and tournament standings. Based on their functionality, it would seem these have extremely simple schemas.
In some MMORPGs, inventory is handled in the obvious way ?each item has an associated record. But Guild Wars does things differently. Everything about a character is one big BLOB (Binary Large OBject), typically 10K-30K in size. (One component is the whole game map, or at least a record of which pixels the character has opened up.) More precisely, there a character BLOB and an account BLOB. BLOBs are updated as the game progresses, then saved every few minutes.

Actually, if there are are 150,000+ transactions per minute, and each character generates a transaction every few minutes, that suggests there are ?million or so characters online at once, peaking at 1 million. Based on what I know about gameplay and so on, I think that figure is too high by a factor of 10 or so. I cant immediately explain the discrepancy.

Of course, there抯 almost no simpler kind of transaction than banging a BLOB into a database. Even so, thats lot of BLOB-banging. So between the game server and the database backends, there are custom cache servers. The game servers send messages about various kinds of structured game-object transactions to the cache servers; the cache servers then bang updated BLOBs into the database. These cache servers are extremely stable, staying up 150-200+ days at a time.

ArenaNet is, on the whole, quite responsive to player requests for upgrades. But in one regard they e consistently disappointed people ?they have omitted introducing an ingame auction house, something competitive MMORPG makers seem to be able to build. Even after the interview, I not 100% clear on the reasons, but I think they mainly boil down to this ?an auction house would be by far the most complex transactional system they e implemented. And given that the inventory isn currently handled on a record-by-record basis, it pretty much would have to be built from scratch. That said, while they gave me no hint of this ?when they reexamine the issue for Guild Wars 2, I bet they抣l realize it not as hard as they now think.

As for other database futures ?just taking account (as oppose to character) inventory from 20 to 80 slots required database scale-out. More database scale-out in the future is likely, although they haven firmly decided on that yet. ArenaNet seems very happy with SQL Server, and seems unlikely to fix what isn broken; besides, they e based in Bellevue. (By way of contrast, Sony Online is in the process of moving Everquest from Oracle to Enterprise DB. I hope to interview them in the future.) ArenaNet doesn seem to be considering memory-centric technology, although that seems an obvious choice for caching; in particular, StreamBase seems to be getting a little traction in the game/virtual world market. (More on that later if the StreamBase folks help me get some interviews.)

Want to continue getting great research about cutting-edge technology, from hardcore IT to wild-and-crazy internet services? Get a FREE subscription by RSS/Atom or e-mail! We recommend taking the integrated feed for all our blogs, but blog-specific ones are also easily available.