G’day Internet. Welcome to Stick Ninjas DevLog 21. My name is Andrew Russell.

What are we doing here? Why aren’t we on YouTube? Where are the pretty pictures?

Well, for the last couple of weeks, since my last DevLog video, I’ve been working on the super-super-low-level bit-packing stuff for the networking in Stick Ninjas. So, currently, the closest thing I have to an entertaining video is this screenshot of my unit-tests completing:

(Oooooohhh! Amazing!)

If you’ve seen the videos, you’ll know that Stick Ninjas, up until now, has been running on a virtual network. You can see the packets flying across that virtual network in the videos, or a screenshot like this one:

While all of the clients (plus the server) are running inside a single application, each has its own viewport and runs essentially an independent instance of the game. Each has an interface for sending and receiving “packet objects”, and those interfaces are linked up by virtual connections that delay and drop packets to simulate a real network.

At the moment my packet objects are – well – kind-of “cheating”.

A real network packet is nothing more than a stream of bytes. My packets are classes that have type information (eg: a “Snapshot” packet and a “Client Input” packet) and named fields (eg: “tick number”). While I do serialize each actor into a byte stream, each actor gets its own separate byte stream that gets put into a lookup table.

This has been great for prototyping. But now it’s time to get those packets encoded for use on a real network. This involves solving several problems:

First of all, there’s the format of the packets themselves. Information, like: the type of each packet, where each actor’s data starts in the packet, where the event data starts (see DevLog 18), and so on; must be encoded into a flat, binary representation.

Next, the actual encoding must be done in as little space as possible. The smaller I can make the data representation, the smaller each packet will be, and the more clients a server will be able to support for a given connection. To achieve this, I’ve implemented bit-packing for many different kinds of data.

For example: a direction is best represented as a 2D vector, which is stored as two 32-bit floating-point numbers, for a relatively enormous total of 64 bits. But, for the purpose transmitting it over the network, a direction could be stored as a fixed-point angle in just 8 or 9 bits!

(8 bits allows for encoding of 256 possible angles, 9 bits allows for 512; more than enough for most purposes.)

While most data can’t be packed quite so tightly, I’m still expecting bit-packing to reduce the space required for actor data by around 75%.

Finally, there’s delta-compression.

If you look at the above screenshot again, you’ll notice that the server is maintaining a “Snapshot History” and each client has a buffer of received snapshots (visualised in the ruler-like display along the bottom of each viewport).

Along with player input, each client transmits back to the server an acknowledgement of the latest-received snapshot. The server tracks this and, rather than sending a full snapshot, the server can send the difference between this most-recently-acknowledged snapshot and the current one.

This “diffing” is done bit-by-bit across the entire snapshot, with special handling to allow it to work across variable-length data – like if an actor is added or deleted somewhere in the middle of the actor list, in the time between the two snapshots being diffed.

This means that actors that don’t change between snapshots are compressed to take up almost no space. As are individual properties that don’t change. And even parts of properties that don’t change get compressed! For example: say a player’s position changes – an example binary representation might be a change from 01111100 to 01111101 – most of the bits remain the same, and those can be encoded as a (much smaller) instruction to “copy these bits from the previous snapshot”.

~

So that’s what I’ve been working on for the last three weeks while I’ve been neglecting to post a video. (Sorry!)

(That, and it’s currently the middle of summer here in Australia. And making a video while I’m all hot and sweaty isn’t really appealing.)

There are a few reasons that this section of development is taking so long:

First of all, I must admit that this kind of thing is extremely fun to code. There is a temptation to make the best and fastest data packer and compressor ever. So far I’ve mostly avoided getting lost in this very compelling optimisation puzzle. Mostly.

And, secondly, this rather complicated code must be 100% robust and bug-free.

Most of the time, if I make a mistake, it’s pretty obvious where it’s happened and how to fix it: A collision fails, an actor moves in the wrong direction, something is the wrong colour, etc.

But the network code – I need to be able to completely trust that, if I put data into it, I will get the exact same data back out of it. Every time. Without fail. An error in this code could mean an actor state gets corrupted, or the entire game crashes – or, worse still, the networking takes up more bandwidth than it should, or silently drops packets when it shouldn’t. And these horrible things could happen a long time after the error was introduced, on a completely different machine! There is no simple, visual way to check that this code is working correctly – hence the greater degree of care, and the myriad of unit tests.

This isn’t prototyping any more. This is production-quality code. Which means the Stick Ninjas pre-release is getting closer. Stay tuned!

(No schedule for the next DevLog. Hopefully less than three weeks this time ;))