Potential Games LLC

I spent this week running longer performance tests under heavier load, with the goal of preserving maximum service uptime. The game server runs exceptionally well, so long as no one logs in. Even then, it does really well, until I reach my server's memory capacity. Even at capacity, server CPU load remains low. Nonetheless, I have found a few faults, which cause undesirable behavior in the client and server, under certain failure cases. For the last few days, I've been diagnosing these cases. Improvements have begun, which will spill into next week. Afterward, the client and server should be more resilient to failure cases. Once these improvements are complete, the next group of tasking involves client and gameplay enhancements.

I had inadvertently placed a hex upon my avatar animation sequences when I chose to number the frames in hexadecimal. Several avatar animations, including all of the attack and death sequences, had been cut short. The logic for loading animation frames counts in hex [00,01,...,09,0a,0b,...]. However, the external resources were numbered in decimal [00,01,...,09,10,11,...]. After frame [09], the loading logic didn't find frame [0a] (because it was numbered [10]) and stopped. Consequently, several frames of animation (up to 8 for some death sequences) were being ignored. Now that everything is numbered in hex, the attack and death sequences are a bit more ... animated.

As a respite from server improvements, I've spent most of this week on client-side graphic rendering improvements. The updates are not yet released, as there is one more little item to deal with. Since it's not quite finished, I haven't taken precise measurements yet, but preliminary results look like about 80% speedup to the world painting routine. I like to leave myself a lot of room for improvement. Then again, the client ran rather well even with no pre-rendering optimizations. Every frame of every character animation layered the base form, performed color masking, and added adornments. Now, the animations are pre-rendered for each avatar. The world surface was painted on-demand, including the complicated masking to produce the terrain edging patterns. Now the static parts of the map are pre-rendered to an off-screen image, reducing painting time. The purpose of these improvements is to support larger numbers of on-screen elements (which did cause some slowdown in tests and according to reports from playtesters). Drawing speed improvements are, however, at the expense of a large amount of client-side RAM consumption. In theory, the game could run in either of two modes: low-memory/high-CPU, or high-memory/low-CPU. It's a textbook performance tradeoff. I'll be excited to hear how it performs for playtesters, just as soon as I can release it.

Today, the last of the Internet's available IPv4 address space has been depleted. Potential Games is pleased to announce that potentialgames.com is now reachable via IPv6, as well as legacy IPv4.

Improving server scalability has been the overarching goal of recent tasking. I have not yet definitively measured performance as client count increases, but I've addressed what I believe to be the major performance impediments. This week, I made two important changes. First, I've reduced more server memory demand by consolidating two content types used to manage creatures. Creatures outnumber characters by the thousands. On top of that, each creature content object incurred a client-visible distributed avatar data model. This nearly doubled the RAM required for each creature instance. With minor augmentation, the avatar class now handles everything needed for creatures (as well as player characters, which is their other purpose). Second, I improved protocol broadcast messages. Some protocols used to deliver messages to all connected sessions, rather than selectively to designated clients. This avoided duplicating the backing byte buffer (to save server memory), but was wasteful of bandwidth. Now the low-level network library provides a better mechanism for selective delivery. I always follow the philosophy to code for correctness first, then efficiency, but this should probably have been improved earlier. The scant testing I perform on my development machine already shows improvement. Once these updates are released to the playtesting server, I'll run a number of tests to measure server performance and observe behavior. Next on the docket are client performance improvements.

While profiling the runtime memory allocation in my game server, I found that my Coordinate class had far-and-away the most numerous instances (over 600,000 at one count). Even though small (each Coordinate includes latitude and longitude integers), this was also the largest contributor to live memory in the Java Virtual Machine. The JVM is very good at creating and disposing of small, short-lived objects. I suspect, however, that many of these Coordinate instances are being retained by server subsystems. For example, map logic, entity positions, and fog coverage all retain Coordinate objects. What is more, many of these Coordinate instances are likely to be redundant copies of the same [latitude, longitude] pair, especially because most maps in the world share the same coordinate space. This is exactly the time to employ the flyweight design pattern. The flyweight design pattern avoids allocating redundant objects by sharing instances. Instead of hundreds of Coordinate[12,12] being instantiated, only one is created. This reduces the amount of overall memory allocation, but at some cost. One cost is the bookkeeping (data structure and algorithm overhead) needed to store and look up existing objects. A Coordinate object is requested by its latitude and longitude, which I turn into a unique key. The simplest approach is to maintain a Map, but this has the downside of keeping every instance ever created in memory. To avoid the cost of retaining references to rarely requested Coordinates, I employed a LinkedHashMap with the access-order constructor flag, overriding the removeEldestEntry() method to produce a least-recently-used (LRU) collection. The least requested Coordinate objects are dropped from the collection, based on an arbitrary maximum capacity (which was determined by experimentally observing the profiling at different thresholds). Once the flyweight collection is full, it does incur higher baseline memory allocation, even when the server is idle, because the Coordinate instances cannot be garbage collected. As a further improvement (not yet implemented), the map could store weak references to the Coordinate objects, to avoid keeping them in memory when no longer needed. In practice, the LRU flyweight Coordinate mechanism appears to lower peak memory allocation and gives a more stable memory profile (not so much spiking and garbage collecting). Instead of hundreds of thousands of live Coordinate instances, there are at most tens of thousands held as flyweights and some stragglers that were dropped from the LRU collection. The overall goal was to reduce and stabilize long-term server memory allocation, which appears to be happening.

In my personal obsession to deallocate all unnecessary content from server memory, I was down to one or two Character objects still hanging around after all clients had exited. Notice that these aren't java.lang.Character objects, but instances of my gameplay Character class. The issue was that somewhere some subsystem was holding a (hard) reference to the Character object. I used VisualVM to create a heap dump of the running server. The heap showed a single Character object instance being held by the Pulse subsystem. Further investigation revealed flawed Character exit world logic, which neglected to signal the final departure of a Character in some cases. With the departure signal fixed, I have not been able to reproduce this problem (this is a good thing). When all game clients exit, all content objects are now deallocated. This should result in a more stable server, capable of longer uptime. In addition, this investigation revealed other areas of improvement to pursue. "Always... no, no... never... forget to check your references." -- Dr. Meredith

The ripple effect of last week's memory deallocation strategy were supposed to diminish into a placid calm. Instead, they amplified into waves of memory allocation, crashing against the rocky shores of garbage collection. The particular problem was (is) that my meager test server was not able to ride the choppy seas, churned by a maelstrom of automated test players I unleashed upon it, again and again. I was able to calm the storm through improved (fixed?) content logic. Despite this stirring tale of memory mitigation, there are still some rough seas to navigate. The server is now more aggressive in its memory cleanup, but continues to consume more than is practical. This week's voyage of discovery has charted a number of new islands of improvement to be explored... Check back next week for my continuing adventures in game software development.

Development week 2 of 2011 culminated in a more general approach to server memory management. As shown in last week's post, memory tends to allocate and stay that way. Even after the hordes of playtesters (both of them) have turned in for the evening, server memory remains at high consumption. This makes for some dull graphs: To make the graphs more interesting, the server now unloads gameplay objects (characters, creatures, avatars, and maps) it no longer needs: Better yet, RAM allocation follows suit:

Today's anomaly: Monsters that previously killed themselves (to death) are now resurrecting when the map is restored. I suspect the server must be situated upon a pet cemetery (or there's a logic flaw in the code). *Time passes* As it turns out, there is no (known) pet cemetery. Rather, I was saving the state of dead creatures, but not restoring their deadness when reloaded. To simplify, I'm just not going to persist dead creatures.