Hyacinth's Multithreaded Server Architecture

Overview

During my final year at university, I decided to take a computer networking course as part of my CS minor. While the course taught me the fundamentals behind networked systems, I was still curious as to how they applied to multiplayer online games. This project was the result of diving headfirst into this rabbithole and immersing myself in creating a netcode system, from scratch, capable of supporting a first person shooter game.

One of my all-time favorite game genre is first-person shooters. Their highly skilled and competitive nature is what initially drew me in, and this project helped me learn about the different ways that online games support competitive integrity. A majority of the network architecture decisions I made were based on resources from existing multiplayer games such as Valve's CS2, Riot's Valorant, and Blizzard's Overwatch.

This article assumes a basic understanding of networking principles in C/C++, including creating/binding sockets and sending/receiving packets.

You can view the source code for both client and server at: https://github.com/AKris0090/Hyacinth

Contents

Hyacinth is a multiplayer first person shooter engine written in C++ using Vulkan for graphics and Nvidia PhysX for physics.

This article will go over the architecture and features I implemented in Hyacinth's client & server model, including:

Architecture Breakdown

All of Hyacinth's core server functions occur in parallel thanks to its multithreaded nature. You can see the breakdown of each thread and its assigned task:

UDP Receiver Thread

This thread handles the task of listening for UDP packets, deserializing, timestamping, and placing them in the queue to be processed.

When a client initially attempts to connect to the server, I have it shoot the server a "tryping" message every one second. When the server receives this ping, the receiver thread records the internet address of the sending client, creates a CLIENT_JOIN event, and pushes it to the server event queue. This address will later be used for sending snapshots to.

CHALLENGE: Originally, I used a naive implementation for the client handshake over TCP for its reliability. The client would connect to a TCP socket, which required yet another thread to maintain. The issue with this is that if two clients attempted to connect at the same time, only one would be allowed to connect with the TCP socket, and the other would silently crash. To avoid the issue of having to manage multiple TCP sockets per client, I chose to switch everything over to UDP. By implementing a blocking layer that requires a response from the server to continue execution, I can make sure that the client receives all the information it needs to start displaying server entities/information over UDP.

Watchdog Thread

Each entity is assigned a heartbeat marking the last moment in time the server received a packet from the client. This thread is responsible for monitoring those heartbeats and detecting when a client is no longer "connected". After creating and pushing a CLIENT_DISCONNECT event to the server event queue, it also interfaces with the physics system to remove the client's corresponding character controller from the simulation. The thread sleeps for a full second between checks.

Tick Update Thread

This is the "main" thread, acting as the owner of the entities managed by the server. It fires at a rate of 128hz - every 7.8125 milliseconds.

Every loop, it starts by draining the event queue - adding and removing clients as needed. Then, it drains the packet queue filled by the UDP receiver thread, where it hands out packets to their respective clients (almost like a mail delivery service). It also updates the client heartbeat.

Next, it loops over each client and loads the packet it intends to use for the current tick into its bufferedPacket member. If the client does not have a packet for this tick, it uses the previously applied packet (extrapolation). Using these buffered packets, it calls for a physics simulation update.

It then loops over each client, checking if the client is A) attempting to shoot and B) allowed to shoot by game logic. If the client passes, it calculates the rewinded tick of the client, determined by its ping, retrieves the snapshot (if the rewinded tick is not too far in the past), and passes the client's entity and the server snapshot to the physics manager to perform raycasts.

Finally, it sends out a personalized server snapshot to each client. It timestamps the exact moment when the message was sent out to be used for future ping calculations.

Synchronization + Physics

Modern video games use fixed timesteps for physics due to "simulation divergence"; a physics simulation running at a rate of 30hz will eventually diverge from one running at 128hz. This can be solved by decoupling the physics from the render update in the engine and running it at a stable frequency. This means that a client rendering at a frame rate lower than 128hz might see more than one physics step between frames. On the other hand, a client rendering at a frame rate higher than 128hz would have to linearly interpolate between two physics states to display smoothly and correctly. See the section on interpolation for more info. This consistently timestepped physics simulation allows us to achieve the same results between the client and the server simulations. It further enables implementation of client prediction and server reconciliation.

One of the biggest issues with networked games is the unpredictability of the network. Hyacinth uses the UDP protocol to send messages, which means packets often arrive at the server late, early, or not at all. This means that we need to establish some form of synchronization between the inputs as they are sent by the client and received by the server.

There are multiple ways to go about this, but I chose to go with the solution presented in Valorant's netcode article . On the first packet received by the server from each client, the server establishes a mapping between the client's local reported tick number and the current tick number being processed; by increasing or decreasing the width of the mapping, the server is able to keep up with spikes in packet latency and ensure that the buffer of inputs is always full. By default, this mapping is offset by 2 fixed frames, acting as a buffer for network jitter (packets arriving too early / late). When reporting snapshots back to each client, the server marks the packet with the current tick number translated back into the client's local time frame, using the same mapping from before.

In this example, Client A's tick 162 maps to server tick 457, with a tick basis of 295. Similarly, Client B's tick 345 maps to server tick 456, with a tick basis of 111.

You can also see that with the jitter buffer in place, buffered packets that arrive out-of-order (Client A's 162 and 163) are able to queue in the correct order before they are consumed.

Entity Interpolation

When the client receives information from the server about the position of other entities, it must update its locally stored positions to match. However, if the client were to immediately set the entity transforms to those received, that would result in "snappy" movement, as usually the client framerate is higher than the rate at which it receives network updates. To mitigate this effect, the client stores the last two updates it received from the server and linearly interpolates between them to provide smooth motion for all networked entities.

You can see it clearly in this example, where the server timestep is exaggerated to 100ms for demonstrative purposes:

Without interpolation

With interpolation

Note: this is also used with the decoupled physics simulation in client prediction. For general use, I created this thread-safe data structure: 

class InterpolationPacketBuffer {
public:
	float timeAggregate;
	std::pair<ServerSnapshot, ServerSnapshot> packetBuffer;
	std::shared_mutex packetBufferMutex
	void newPacket(ServerSnapshot snap) {
		std::unique_lock<std::shared_mutex> l(packetBufferMutex);
		packetBuffer.first = packetBuffer.second;
		packetBuffer.second = snap;
		timeAggregate = 0.f;
	}

	ServerSnapshot getInterpolatedSimPacket(float timeDelta) {
		timeAggregate += timeDelta;
		if (timeAggregate > SERVER_TIMESTEP) {
			timeAggregate = SERVER_TIMESTEP;
		}
		float alpha = timeAggregate / SERVER_TIMESTEP;
	
		std::shared_lock<std::shared_mutex> l(packetBufferMutex);
		ServerSnapshot p;
		for (auto& fromEntity : packetBuffer.first.entities) {
			Entity* secondEnt = nullptr;
			for (auto& toEntity : packetBuffer.second.entities) {
				if (toEntity.id == fromEntity.id) {
					secondEnt = &toEntity;
				}
			}
			if (!secondEnt) continue;
			Entity newEntity;
			// copy entity fields from 'fromEntity'
			p.entities.push_back(newEntity);
		}
		return p;
	}
};

The getInterpolatedSimPacket() function is called every frame, to update the timeAggregate member. This results in smooth entity movement regardless of client frame rate.

Client Prediction + Server Reconciliation

Under an authoritative game server, each client sends its inputs to the server, and the server responds with each clients' final state. If you factor in packet latency into this scenario, it might be tens or even hundreds of milliseconds of delay between when you press an input and see a result on the screen. For this reason, we run a local physics simulation on the client ahead of the server by half of the packet round trip time (RTT / 2). This concept is known as client prediction, and makes it so that user inputs have an immediate visual result.

However, the server is the authority in this situation. When the server simulates the same inputs as the client, there is a chance that they don't match, and the client must correct itself. This can happen for a variety of reasons, such as if a status effect was applied or if the client took fatal damage. This is known as "server reconciliation".

In order to achieve this, on the client side, we keep track of the pitch, yaw, and movement inputs for each tick that has not yet seen a corresponding confirmation from the server: 

struct StateStorage {
	uint32_t tickNum;
	Transform state;
	int8_t fb, lr; // movement inputs
}

class RewindBuffer {
public:
	using PhysicsPosFn = std::function<void(glm::vec3 p)>;
	using PhysicsStepFn = std::function<void(Transform& t, int8_t fb, int8_t lr)>;
	std::deque<StateStorage> ringBuffer;
	std::queue<ServerSnapshot> pendingPackets;
	std::mutex pendingPacketsMutex;
	std::shared_mutex rBMutex;
	
	void setPhysicsPosition(PhysicsPosFn fn) {
		physicsPosition = fn;
	}

	void setPhysicsStep(PhysicsStepFn fn) {
		physicsStep = fn;
	}

	void addState(Transform t, int8_t fb, int8_t lr, uint32_t tickNum) {
		std::unique_lock<std::shared_mutex> lock(rBMutex);
		StateStorage sS;
		sS.tickNum = tickNum;
		sS.state = t;
		sS.fb = fb;
		sS.lr = lr;
		ringBuffer.push_back(sS);
	}

	std::pair<Transform, Transform> rewindState(Transform newTransform, uint32_t tickNum);
	bool checkPacketNeedsRewind(Entity* self, std::pair<Transform, Transform>& outTransform, Transform serverTransform, uint32_t processedTickNum); // sp.processedTickNum is translated to client local tick

private:
	PhysicsStepFn physicsStep;
	PhysicsPosFn physicsPosition;
};

While going through the packets from the server, we must first figure out if the snapshot necessitates a rewind. We check this by comparing the difference in world position against an epsilon value big enough to ignore floating point precision errors, but small enough to detect real discrepancies. 

bool RewindBuffer::checkPacketNeedsRewind(Entity* self, std::pair<Transform, Transform>& outTransform, Transform serverTransform, uint32_t processedTickNum) {
	while (!ringBuffer.empty() && ringBuffer.front().tickNum != processedTickNum) {
		ringBuffer.pop_front();
	}

	if (ringBuffer.empty()) return false;

	// find corresponding index of the state for processedTickNum in ringBuffer

	if (glm::length(ringBuffer[stateIndex].state.position - serverTransform.position) > DIFF_THRESHOLD) {
		outTransform = rewindState(serverTransform, processedTickNum);
		return true;
	}

	return false;
}

If a rewind is deemed necessary, the client first sets its transform to that reported by the server. It then walks sequentially down each remaining state in the ringBuffer - first setting the client's rotation and then advancing the local physics simulation via function objects. 

std::pair<Transform, Transform> RewindBuffer::rewindState(Transform newTransform, uint32_t tickNum) {
	std::shared_lock<std::shared_mutex> lock(rBMutex);
	physicsPosition(newTransform.position);

	Transform prev;

	bool skip = true;
	for (int i = 0; i < ringBuffer.size(); i++) {
		StateStorage& currentState = ringBuffer[i];
		if (currentState.tickNum == tickNum) { skip = false; currentState.state = newTransform; continue; }
		if (skip) continue;
		newTransform.pitch = currentState.state.pitch;
		newTransform.yaw = currentState.state.yaw;
		newTransform.setRotationPitchYaw();
		physicsStep(newTransform, currentState.fb, currentState.lr);
		currentState.state = newTransform;
		if (i == ringBuffer.size() - 2) {
			prev = newTransform;
		}
	}

	return std::pair<Transform, Transform>(prev, newTransform);
}

Note: the rewind function returns a pair of transforms. This is due to entity interpolation, where the client's transform at any moment is an interpolation of the two most recent past states.

But why? Why do we need to reapply and re-simulate the inputs?

Remember from earlier, the client is always RTT/2 ahead of the server. When a client's input packet arrives at the server, there are an X number of input packets "in flight", where they have been sent by the client, but not yet received by the server. When the client receives a correction from the server and sets the reported transform, it is now in the past. Those "in-flight" inputs were effectively discarded by the client's local simulation, and as they arrive at the server to get simulated, it forms a cascade of continuous rewinds, keeping the client in the past. In order to mitigate this effect, the client must re-simulate all inputs to restore its position in the future.

Lag Compensation

One issue with client-side prediction is that while the player is being simulated in the future, it sees all other entities in the past. If a player aims and shoots at another player, by the time that shot input arrives at the server, the other player might have moved away from that position. To counter this, we store a buffer of state snapshots on the server. The maximum length of this "rewind buffer" is arbitrary, but I set it to 140ms, which is what Valorant has it set at. Then, to calculate the tick that the client saw when it fired a shot, we use this formula, modified from Valve's networking article:

Rewind Tick = Current Tick - Jitter Buffer - (Client's Packet Latency / Ms Per Tick)

While there are many different ways to determine client packet latency, I implemented a system where snapshots get timestamped as soon as they leave the server. I also added a field to the client input packet called "ack", which contains the server tick of the most recently received snapshot. When the server gets a client's packet, it immediately records the current timestamp, then uses the ack field to index into the snapshot/send timestamp queue. By subtracting the send timestamp from the received timestamp and dividing by 2, the server calculates the client's ping for that packet in particular.

If the rewound tick is not too far in the past, it is found in the snapshot buffer and passed to the physics manager along with the shooter's current transform. Using PhysX's geometry query system, I loop over every other entity in the scene and test against a capsule geometry at the entity's current position. The function returns a hitReg struct that states if the shot hit an entity, and which entity got hit. 

hitReg PhysicsManager::playerShooting(uint32_t shooterId, Transform& currentEntityTransform, rewindSnapshot* snapshotToTrace) {
	hitReg h = hitReg{ false, INT_MAX };

	// return hitreg struct that reports if the shooter hit anything, or if hit nothing
	physx::PxVec3 origin = physxVec(currentEntityTransform.position + glm::vec3(0.f, 1.85f, 0.f));
	currentEntityTransform.setRotationPitchYaw();
	physx::PxVec3 dir = physxVec(glm::normalize(currentEntityTransform.forward));
	physx::PxReal maxDist = 100.f;
	physx::PxRaycastBuffer rayHit;

	for (const auto& e : snapshotToTrace->entityPositions) {
		if (e.id == shooterId) continue; // if the current entity is the shooter, skip

		physx::PxTransform pose(physxVec(e.pos + glm::vec3(0, 1.f, 0)), physx::PxQuat(physx::PxHalfPi, physx::PxVec3(0, 0, 1)));
		physx::PxRaycastHit hit;

		bool didHit = physx::PxGeometryQuery::raycast(origin, dir, capGeom, pose, maxDist, physx::PxHitFlag::eDEFAULT, 1, &hit);

		if (didHit) {
			h.hit = true;
			h.entityHitId = e.id;
			break;
		}
	}

	return h;
}

This system makes shooting feel rewarding and accurate, regardless of network latency.

CHALLENGE: One small PhysX quirk that tripped me up is that physX's scene queries use cached representations of the scene to test against. This means that even if you move an actor, you would have to call "flushQueryUpdates()" before running a scene query. On top of this, a character controller has another layer of deferred synchronization to its cached state. Instead of dealing with a parallel physx scene, I simply store the geometry shape for the character hitbox and run a geometry query instead, which allows you to pass in a world transform to position the collider with before it is tested against.

In the example below, the shooter has an exaggerated latency of 700ms to the server. The blue hitbox represents the position of the other player as the client sees it, and the red hitbox represents the position of a server-acknowledged hit. You can see that when the shooter's input arrives at the server nearly a full second later, it is still able to find the other player's past position and reward an accurate shot regardless of latency.

Lag Compensation with 700ms simulated packet latency

Wrapping Up

Learning about the netcode from games like CS:GO, Valorant, and Overwatch has taught me a lot about the underlying systems that maintain competitive integrity regardless of how powerful a player's setup is or their connection quality. There is a ton of engineering work involved in simple networked interactions that many people (including me, before this project) would just take for granted.

One of the biggest takeaways I got from this project was designing and implementing multithreaded systems. Managing shared resources between the core render loop and the decoupled physics simulation was a huge challenge, and I'm learning more about the different types of locks, mutexes, and synchronization structures each day I work on it.

From writing an engine to showcase a DDGI system, to implementing:

... this project has come a really long way. I intend to keep building on it, starting with a major graphics overhaul. More soon!

Resources Used