One field in a control-plane response restarted a live game server every 21 seconds

• What actually happened
• The general lesson
• What changed

Floptle Cloud's dedicated servers run on fleet boxes. Each box runs an agent that polls the control plane's /desired endpoint, writes a systemd unit per server from the answer, and restarts a server whose unit text changed. The engine had just gained a --lobby-code flag so a restarted server could reclaim its old six-character code instead of minting a new one, and I wired lobby_code into /desired so the agent would pass it.

The live server stopped two minutes later.

What actually happened#

My first diagnosis was that the server rejected the unknown flag. It did not. The truth is worse: it started fine. The agent turns lobby_code into a command-line flag on the engine binary the deployment's bundle pins, and the live deployment pinned an engine two releases older than the one that learned the flag. That engine could not reclaim, so on every start it asked the relay for a code and the relay minted a fresh one.

Then the loop: /desired sends code X. The agent writes the unit and restarts the server. The relay mints code Y. The server reports Y. The control plane adopts Y as the deployment's reservation. /desired now sends Y. The unit text changed, so the agent restarts the server. Repeat every 21 seconds. After four rewrites systemd's start limit tripped, the row went to failed, dropped out of /desired, and the agent correctly stopped it.

Zero players were connected. Recovery was reverting the field and putting the row back to starting; the server was up again in about two minutes.

The general lesson#

Anything /desired derives from state that a restart itself changes is a feedback loop. And a field added to /desired is a flag on somebody else's process, whose version is pinned per deployment by the bundle and is routinely older than the agent. Every other field in that response had been understood by every engine that ever read it, which is exactly why it was possible to forget.

What changed#

Two guards, because a version check alone would leave the loop live for any newer server whose reclaim ever failed:

1. lobby_code is withheld unless the bundle pins an engine at or above the release that understands it. Unknown reads as no, and a prerelease of the minimum is below the minimum.
2. If the control plane asked a server to reclaim X and the server reports Y, the reclaim failed, and the reservation is kept rather than overwritten. Adopting looks helpful and is catastrophic. Keeping the promise is also the stable answer: the unit text stops moving and the failure becomes a loud log line instead of an outage.

The incident also exposed that a deployment had been accumulating one held reservation per code it was ever given (five in two minutes), each of which the relay must never mint for anybody, so each was leaked out of the region's namespace. A deployment holds exactly one code now.

Later the same week, with the engine, relay and agent all on releases that understood the flag, the same server reclaimed its code across a restart with players kept, which is what the field was for.