Engineering log

What broke, what we measured, what changed

Notes from running our own systems, dated, with the numbers. Each one was written by the person who found the problem and fixed it. Nothing here is a case study; it is the log.

• 2026-09-11
  2 min read

  I took the relay down for forty seconds, and the pre-flight test had proved nothing

  Putting a real TLS certificate on the Floptle Cloud relay crash-looped it on a permissions error that a test under sudo could not have caught.

• 2026-09-11
  2 min read

  The relay's ceiling was a 208 KB socket buffer, not the link

  A load test on the Floptle Cloud relay found packet drops at around 100 players, using under 1% of the link. Three counters agreed on why.

• 2026-09-10
  2 min read

  One field in a control-plane response restarted a live game server every 21 seconds

  Adding lobby_code to the fleet's desired-state endpoint created a feedback loop with a server whose engine did not understand the flag. Systemd ended it.

• 2026-09-10
  3 min read

  234 kbps per player, and the bug the first real match found

  The first real multiplayer session in Floptle's history, across the Atlantic, gave the number every ceiling is built on and found a bug in ten minutes.

• 2026-09-03
  3 min read

  Finding a failing RAM module through a corrupt Docker image

  A 46-hour analytics outage traced through a ClickHouse checksum failure, a byte-compare against the registry, and a bit pattern only bad memory makes.