Since 2026-09-22, about 1 in 3 of our Google Drive backups fail to upload. This happens for both database backups and file backups, on all of our servers (6 servers).
What happens
- The Google Drive API returns HTTP 502.
- The backup status stays "active", and "Last backup" is updated.
- No failure notification is sent. (Notifications work for other failures. We received "Backup ... has failed" for a different error earlier this month.)
- The upload is not retried. No file reaches Google Drive.
How we found it
The server log (Server → Logs, or GET /api/servers/{server}/logs) for the backup run contains Google's "Error 502 (Server Error)" HTML page. A successful run contains the usual "kind": "drive#file" JSON.
From 2026-09-21 to 2026-09-25 we counted 17 errors in 59 backup runs.
Expected
- A non-2xx response from Google Drive marks the backup as failed and sends a notification.
- The upload is retried with backoff on 5xx errors, for both database and file backups.
Google's Drive API documentation says 5xx errors should be retried with exponential backoff: https://developers.google.com/workspace/drive/api/guides/handle-errors
The post "The source control update" (https://ploi.io/news/the-source-control-update) says database backups have a try limit of 5 and notify on each failure. This does not seem to happen for this case.
We can share log IDs privately if that helps.
-
Wataru Tamanaha moved item to project Bugs
18 hours ago -
Wataru Tamanaha created the item
18 hours ago