The launch of River City Ransom: Underground (Now available on Steam for $19.99!) went pretty well, all things considered. (This isn’t a full post-mortem, but could perhaps be considered a narrow sliver of one.)

We had a few nasty bugs crop up on day 1, and we fixed the worst of them right away. It’s now been two weeks, and we’ve just rolled out the much larger fix for the infamous “multiplayer save bug”.

If you follow me (@_AndrewRussell) or the game (@rivercityransom) on Twitter, you’ll know that the change set for this clocked in at around 3.5 thousand lines of code. This represents about 1% of our total source code (and close to 2 weeks of my time). It really is a massive change.

This post is to explain how such a difficult-to-fix bug ended up being shipped in the first place.

First we need to start with a small piece of technical information: In a networked game, the amount of data we can add to the game state with a joining player is tiny. Maybe a few hundred bytes at most. When I wrote that part of the netcode, I inserted a warning when the size exceeded 64 bytes.

(Details: Our game is P2P and supports drop-in-drop-out at any time, including support for host-migration. This means that player join data can be buffered, replayed, and re-transmitted almost arbitrarily. This starts to become an issue if multiple join and leave events happen in quick succession, particularly across a host-migration.)

The reason this “player-join-data” feature was added to the netcode was specifically so we could allow players to bring their list of unlocked characters into network games, play as them, upgrade their stats, and unlock new characters. The further intention was that players could keep those changes across different network games, as well as carry them into single player.

Unfortunately, the amount of data required by the unlock list was significantly more than was safe to transmit when joining (kilobytes). And, even with substantial data packing work, impossible to fit inside the 64-byte warning limit I had set much earlier in development.

(Additionally, there is a limit on the total size of the “live” game state. And having four full unlock lists in game would have exceeded it.)

This left two other options: Simply use the unlock list of the host (the same as couch play); or allow the joining player to bring one of their characters into the game with them. For the vast majority of development, we went with the first option – as it required zero extra implementation effort. But we did keep the second option in the back of our minds, to perhaps implement when development time became available.

Then, about 4 days before we shipped, Daniel decided that this feature – in the form of the second option – was important enough for us to add at the last minute. (Meanwhile I was coming down with an extremely bad cold, made worse due to the stress and crunch time of launch.)

This was obviously too close to launch to do sufficient testing, or even to properly consider the ramifications of this change. This was further exacerbated by the fact that the change was purposely left out of patch notes (there were 3 more beta releases after the feature was added), explicitly because there were some unfixed bugs in the initial implementation. I am hazy as to whether the bugs were actually fixed before launch, but suffice to say the change never ended up being included in any beta patch notes.

Which brings me to the central thesis of this post: More is Less. By adding this feature, the menu flow when joining an online game was changed from this:

To this:

Now players were selecting a save file when entering an online game. So their expectation was that any progress they made in the online game would be saved to that save file.

So, despite the fact we had added a feature to the game (the ability to play online as a character from your save), it was rightly perceived by players as a critical data-loss bug!

During the beta, where we did not present the save screen, no one batted an eye at the fact that we didn’t save characters in online play. But, after launch, with hundreds of players encountering the new menu flow, it immediately became our number one user complaint. (Even more than any of our actual data-loss bugs.)

So I have spent the last two weeks doing little else besides rewriting the game’s unlock system: Each character can now have their own unlock list, local multiplayer characters will share a list, that entire list comes with you into an online multiplayer game, and it is always saved back into the save file that it came from, even online.

Most of the work involved fixing the many systems that were written with the assumption that there was only a single game-wide unlock list. But there was also a great deal of work packing the data, so that the unlock list could be sent across the network at a reasonable size (238 bytes with all unlocks and no inventory). Plus I’ve fixed all of the bugs around unlocks and save games that I could find in the process.

River City Ransom: Underground just got significantly better. Especially for multiplayer. Now would be a great time to go and play it!