Tailscale traces six-month database outage to 16-year-old SQLite flaw
The open-source networking firm deployed SQLite version 3.51.3 to resolve 19 corruption events, following a complex forensic investigation into aggressive manual checkpointing practices.
Tailscale engineers, in collaboration with core SQLite developers, have identified a 16-year-old data race bug in the SQLite database engine as the primary cause of significant instability across the company’s control plane. The flaw, designated as the WAL-Reset bug, triggered 19 separate database corruption incidents over a six-month period, disrupting service reliability and eroding user trust. The issue was resolved following the deployment of SQLite version 3.51.3.
The corruption events originated from a data race between a checkpoint operation and a write transaction. Tailscale’s infrastructure utilises a non-standard configuration involving aggressive manual checkpointing to support fast, consistent backups. This approach caused the checkpointing process to incorrectly assume pages had been copied from the Write-Ahead Log to the main database file, resulting in permanent data loss and subsequent corruption of the database file.
Isolating the root cause required extensive forensic telemetry. After initial attempts to reproduce the bug synthetically failed, the engineering team implemented a transaction logging pipeline to stream SQL statements. This revealed that committed data was occasionally invisible to later transactions. The SQLite developers subsequently created a custom tmstmpvfs shim to trace virtual filesystem changes, which allowed them to pinpoint the race condition within the checkpoint logic.
The resolution process encountered a secondary complication during the rollout of SQLite version 3.52.0, which contained the fix for the WAL-Reset bug. Tailscale discovered a stale expression index bug in this version, where changes to rounding behaviour for text-to-floating-point conversions triggered false corruption warnings. Consequently, version 3.52.0 was withdrawn, and the company deployed the stable version 3.51.3 instead.
Post-deployment monitoring confirmed the efficacy of the fix. Two months after the patch was applied, a warning alert fired, proving that the precise conditions for the WAL-Reset bug were actively occurring in the production environment but were being successfully prevented by the patch. Tailscale has since funded the open-source VFS shim to aid future bug detection and refined its database backup and recovery processes to mitigate similar risks.
