Potential Games LLC

Running autonomous test clients against my game server, I kept generating a situation in which several client network connections (TCP) remained open. This resulted in those client sessions remaining active long after the client had disconnected. This had the potential to retain game state in the server, preventing memory from being reclaimed (effectively a memory leak). The player might not be able to log in later, because they still have an active (albeit bogus) session. No matter how many safety checks I put into the networking logic, I could not detect these broken, lingering connections. As it turns out, Java sockets cannot report they have closed when the socket was not shut down cleanly, without actively attempting to read/write data. My game server protocols are rather conservative, and do not chatter unnecessarily with clients. As a result, an abruptly disconnected client socket could idle indefinitely, so long as no data was destined to be sent its way. To alleviate this problem, the server networking logic now takes note of client communication times. After a configurable timeout with no activity, the server sends a ping, to which the client is to pong. This has one of three results. If the pong is not returned within another probationary timeout, the client is considered lost, and the session is forcibly closed. If the client is, in fact, alive and well, the pong refreshes its most recent activity. The third result is what I've most commonly found: the attempt to send the ping over a broken connection triggers a network error (IOException), which is caught and handled, more-or-less, cleanly. In a perfect world, client software would cleanly tear down all TCP connections. In the real world, several factors can prevent this ideal behavior (network failure, software crash, killed application). In any case, server systems cannot rely on clients to behave ideally. While I could spend the rest of my days improving network communication logic (resume available upon request), the above technique is simple enough and appears to be working well in ongoing tests.

Remember those client/gameplay updates I've been promising... I'm almost ready to get going on that. This week I've been hunting down excessive server memory allocations and deflating them. The goal is to have a predictable (and low) memory profile, so I can gauge the RAM required to support N players (as N approaches infinity, if all goes to plan). The particular improvements include content persistence, atlas activities, and server startup data indexing. These activities were each consuming large chunks of memory, which could spike and cause OutOfMemoryErrors. One technique I used to lower runtime RAM consumption was to stream content directly to the backing store during persistence, rather than constructing large intermediate data structures in memory. Atlas maps are served up directly from disk, when possible, rather than redundantly inflating/deflating in memory. Startup data indexing now builds its lookup structures on disk, rather than in memory. Overall, this should make for a smoother ride. Although I could spend the rest of my days improving system software design and implementation (resume available upon request), I'm eager to re-focus on the player experience with client and gameplay enhancements. Why can't there be N of me?

I had inadvertently placed a hex upon my avatar animation sequences when I chose to number the frames in hexadecimal. Several avatar animations, including all of the attack and death sequences, had been cut short. The logic for loading animation frames counts in hex [00,01,...,09,0a,0b,...]. However, the external resources were numbered in decimal [00,01,...,09,10,11,...]. After frame [09], the loading logic didn't find frame [0a] (because it was numbered [10]) and stopped. Consequently, several frames of animation (up to 8 for some death sequences) were being ignored. Now that everything is numbered in hex, the attack and death sequences are a bit more ... animated.

As a respite from server improvements, I've spent most of this week on client-side graphic rendering improvements. The updates are not yet released, as there is one more little item to deal with. Since it's not quite finished, I haven't taken precise measurements yet, but preliminary results look like about 80% speedup to the world painting routine. I like to leave myself a lot of room for improvement. Then again, the client ran rather well even with no pre-rendering optimizations. Every frame of every character animation layered the base form, performed color masking, and added adornments. Now, the animations are pre-rendered for each avatar. The world surface was painted on-demand, including the complicated masking to produce the terrain edging patterns. Now the static parts of the map are pre-rendered to an off-screen image, reducing painting time. The purpose of these improvements is to support larger numbers of on-screen elements (which did cause some slowdown in tests and according to reports from playtesters). Drawing speed improvements are, however, at the expense of a large amount of client-side RAM consumption. In theory, the game could run in either of two modes: low-memory/high-CPU, or high-memory/low-CPU. It's a textbook performance tradeoff. I'll be excited to hear how it performs for playtesters, just as soon as I can release it.

While profiling the runtime memory allocation in my game server, I found that my Coordinate class had far-and-away the most numerous instances (over 600,000 at one count). Even though small (each Coordinate includes latitude and longitude integers), this was also the largest contributor to live memory in the Java Virtual Machine. The JVM is very good at creating and disposing of small, short-lived objects. I suspect, however, that many of these Coordinate instances are being retained by server subsystems. For example, map logic, entity positions, and fog coverage all retain Coordinate objects. What is more, many of these Coordinate instances are likely to be redundant copies of the same [latitude, longitude] pair, especially because most maps in the world share the same coordinate space. This is exactly the time to employ the flyweight design pattern. The flyweight design pattern avoids allocating redundant objects by sharing instances. Instead of hundreds of Coordinate[12,12] being instantiated, only one is created. This reduces the amount of overall memory allocation, but at some cost. One cost is the bookkeeping (data structure and algorithm overhead) needed to store and look up existing objects. A Coordinate object is requested by its latitude and longitude, which I turn into a unique key. The simplest approach is to maintain a Map, but this has the downside of keeping every instance ever created in memory. To avoid the cost of retaining references to rarely requested Coordinates, I employed a LinkedHashMap with the access-order constructor flag, overriding the removeEldestEntry() method to produce a least-recently-used (LRU) collection. The least requested Coordinate objects are dropped from the collection, based on an arbitrary maximum capacity (which was determined by experimentally observing the profiling at different thresholds). Once the flyweight collection is full, it does incur higher baseline memory allocation, even when the server is idle, because the Coordinate instances cannot be garbage collected. As a further improvement (not yet implemented), the map could store weak references to the Coordinate objects, to avoid keeping them in memory when no longer needed. In practice, the LRU flyweight Coordinate mechanism appears to lower peak memory allocation and gives a more stable memory profile (not so much spiking and garbage collecting). Instead of hundreds of thousands of live Coordinate instances, there are at most tens of thousands held as flyweights and some stragglers that were dropped from the LRU collection. The overall goal was to reduce and stabilize long-term server memory allocation, which appears to be happening.

In my personal obsession to deallocate all unnecessary content from server memory, I was down to one or two Character objects still hanging around after all clients had exited. Notice that these aren't java.lang.Character objects, but instances of my gameplay Character class. The issue was that somewhere some subsystem was holding a (hard) reference to the Character object. I used VisualVM to create a heap dump of the running server. The heap showed a single Character object instance being held by the Pulse subsystem. Further investigation revealed flawed Character exit world logic, which neglected to signal the final departure of a Character in some cases. With the departure signal fixed, I have not been able to reproduce this problem (this is a good thing). When all game clients exit, all content objects are now deallocated. This should result in a more stable server, capable of longer uptime. In addition, this investigation revealed other areas of improvement to pursue. "Always... no, no... never... forget to check your references." -- Dr. Meredith

Development week 2 of 2011 culminated in a more general approach to server memory management. As shown in last week's post, memory tends to allocate and stay that way. Even after the hordes of playtesters (both of them) have turned in for the evening, server memory remains at high consumption. This makes for some dull graphs: To make the graphs more interesting, the server now unloads gameplay objects (characters, creatures, avatars, and maps) it no longer needs: Better yet, RAM allocation follows suit:

This week, I was distracted by my Internet connection failing regularly. A few hours on the phone and a couple cable modems later, I think I'm stable once again. None of this affects the test server, which has been running smoothly, even though it's (meager) memory is nearly fully allocated. Lately I've been focused on server and network performance. I've already significantly reduced bandwidth requirements for the server. Memory allocation needs some tuning, as many content objects can't be entirely deallocated at runtime yet. I'm just wrapping up another change, which will improve network performance, as well as lessen the overall burden on the client. I won't get into client-specific (e.g., graphic rendering) performance in earnest until I've addressed the remaining server memory allocation issues (next week).

I recently delved into the java.lang.management package, in order to measure thread activity in my Potential RPG server application. Tools like VisualVM offer real-time probing, but I also like to have my applications produce their own statistical results. In this case, I'm logging CPU time (and a bunch of other stats) from my server application, which I use to produce performance charts (using the JFreeChart library). In java.lang.management, OperatingSystemMXBean reports the CPU time of your application process. The ThreadMXBean class reports the CPU time of each active thread. CPU time measures time spent being executed by the CPU. Contrast this with wallclock time, which may include time when your process is swapped out for other work. In addition, you can query user time, which is the portion of CPU time spent executing your application, as opposed to performing system time events such as disk I/O. Java tip: How to get CPU, system, and user time for benchmarking is an excellent article on using these classes. In particular, notice that ThreadMXBean does not report on threads that have died. To avoid missing thread activity, it is important to poll for thread activity at some granularity. I'm trying a 500ms polling rate, but my initial results imply that the thread I'm using to do this polling is using a significant percentage of the overall CPU time. Any thread activity missed by the polling granularity can be reported by subtracting the sum of sampled thread time from the total process CPU time. By increasing the polling rate, you can catch more short-lived threads, at the expense of more overall CPU time spent polling. In some cases, I'm using a thread pool (Executors.newCachedThreadPool()). By default, each thread will have a unique name, but this is not a requirement. By passing in a custom thread factory, I can name all threads that perform a similar service the same. In my thread polling routine, I add the CPU time of each thread by name. This way, even if several threads are actively performing some task, the CPU time is reported for the group. By monitoring my thread activity, I've found which parts of my application are working the hardest. I've also identified a few rogue threads (probably quick-and-dirty cases in which I extend Thread itself), which need to be wrangled. These might account for the CPU time not being captured by my polling. Overall, these thread monitoring results give me a good idea where to improve performance.

Mouse events from Java Swing report the click count. When mouse events are generated within some click threshold (which should be an operating system setting), the click count increases. One common problem is processing single- versus double-clicks. Java does not hold back the single-click event (click count of 1) even when a double-click (click count of 2) may be forthcoming. This is for good reason: As well designed of a programming language that it is, Java cannot see the future. There are techniques (documented elsewhere) for dealing with this aspect of mouse processing in Java. The impetus for this article is mouse event inconsistencies between platforms. Particularly, click counts seem to be reported differently for different events on different platforms. Read the rest of this entry »