Finding a failing RAM module through a corrupt Docker image

• The chain
• Why the first tests said the RAM was fine
• What this meant for the rest of the server
• What changed

The symptom was that our self-hosted analytics went dark for 46 hours. The tracker endpoint returned 502, so every visit in that window went unrecorded, and since the tracker posts live with nothing queued, it is gone. The traffic graph for those two days reads as a cliff. It is an outage, not a cliff.

The chain#

The analytics database container was exiting every sixty seconds with code 246, CORRUPTED_DATA, and had restarted 2,758 times. ClickHouse checksums its own executable at startup, and the binary inside the locally pulled image had flipped bits. The application container then failed on a DNS error because the database was never up long enough to resolve. That DNS error is where a first look leads, and it is a symptom. Chasing it goes nowhere.

To prove the image was wrong rather than the software, I fetched the same layer straight from Docker Hub with curl (token, manifest, blob, sha256 verified against the registry's digest) and byte-compared it with the extracted copy on disk. Thirteen bytes differed out of 499,081,544, every one a single-bit flip, all within a window of about 115 KB. Four fresh pulls each produced a different corrupt copy.

Why the first tests said the RAM was fine#

A userspace memory test ran 48 GiB clean. Filesystem writes and reads, direct-IO cold reads, tmpfs, overlay writes, an 8 GiB disk scan and gzip all came back byte-perfect. The cells are marginal, not dead, so a short clean run is not evidence.

A longer pass caught it: 24 single-bit errors over 96 GiB. The fingerprint is what settled it. Every XOR between expected and observed was 0x01 or 0x80. Only bit 0 or bit 7, never bits 1 through 6, and every bad address fell within roughly 100 KB of each other. That is the same pattern as the corrupt binaries. Random errors look nothing like it.

Big Docker layers are the canary because unpacking one pushes half a gigabyte through memory in one go, so it lands on the bad cells often. A tiny image matched the registry exactly.

What this meant for the rest of the server#

The box is a VM on a host with non-ECC memory, so the bad module is on the host and every guest shares it. That includes the production database, which had data_checksums off, meaning a silently corrupted page would neither be detected nor reported. Database backups pass through the same memory, so a backup taken on that box had to be treated as suspect until restore-tested.

Nothing reported any of this on its own. The guest kernel logged zero memory, ECC, filesystem or I/O events. One grep for "ecc" in the kernel log returned false positives by matching container interface names like veth87b7ecc.

What changed#

Service came back with a verified-good binary bind-mounted read-only over the corrupt one, and 28,326 historical events were intact. Re-pulling the image would have re-extracted it through the same failing memory, so that was the one fix not to reach for.

The host has no display, so a bootable memory tester was impractical. I wrote a small program that finds bad physical pages from the host's own shell via /proc/self/pagemap and prints the kernel memmap= lines to reserve them. Two 16 MiB blocks now cover both bad clusters, and the same test reads zero errors over 960 GiB where it found 107 before. A weekly canary runs in the guest. The quarantine is pinned to physical addresses, so any firmware change invalidates it, and the real fix is ECC memory, which is worth it on a box holding other people's data.