I took the relay down for forty seconds, and the pre-flight test had proved nothing

• The outage
• Why the test before the change proved nothing
• What changed

The relay at us-east.relay.fopull.com needed a real certificate. Until then it minted a self-signed one at startup, which meant anyone on the path could read and replace the traffic of every managed session. The certificate itself was straightforward: Let's Encrypt via DNS-01 through Cloudflare, since the relay answers UDP on 7788 and nothing on port 80. The renewal hook deploys the pair atomically and the relay reloads it without a restart. All of that worked first time.

The outage#

I placed the certificate and key at /etc/floptle/tls/, restarted the relay, and it crash-looped on Permission denied.

The relay runs under systemd with DynamicUser=yes, so it has no fixed user and no home. Its API token arrives through LoadCredential, which systemd reads as root and hands to the service through a private path. That is the point of the mechanism, and it is also why the service had never once traversed /etc/floptle itself. The directory is 0700 root. The relay could not enter it, so it could not read the certificate, so it died, forty seconds at a time, until I reverted the unit. The dedicated server that had been connected to it reconnected and reclaimed its lobby code on its own once the relay was back, both times.

Why the test before the change proved nothing#

I had checked that the files were readable before restarting anything. I checked with sudo cat. Root can read anything, so the test answered a question nobody was asking.

The test that answers "can the service read this" has to run as the service would:

systemd-run -p DynamicUser=yes -p SupplementaryGroups=floptle-tls \
  --wait cat /etc/floptle-tls/fullchain.pem

That is the exact mechanism the unit uses, and it would have failed before the restart instead of after.

What changed#

The certificate pair lives in /etc/floptle-tls/ now, 0750 root:floptle-tls, with the relay's dynamic user granted that supplementary group. The runbook for a second region has the systemd-run check in it as a step, not a suggestion. A simulated renewal later that day succeeded end to end: the hook fired, the relay logged that the certificate had been renewed on disk, and NRestarts stayed at zero.

The bench that verifies the chain reported VERIFIED against a real certificate for the first time, which was the thing the whole day was for.