Under investigation. Updates here as available.
And we’re back.
Still unclear what just happened, but it seems we’re ok for now.
1 Like
Summary:
The Mastodon server began returning “500 - Internal Server Error” messages for all interactions at Nov 07 20:14:38 UTC.
The issue was resolved at Nov 07 20:27:08 UTC.
What Happened:
When attempting to run a database backup the Digital Ocean managed database cluster became unresponsive.
During the outage window, the database server was still allowing connections, but all queries failed.
The server continued to report that the database was empty until the incident end, when everything suddenly began working properly again.
The database backup is a routine activity which has been conducted many times previously without incident.
The behaviour of the database - the apparent, transient loss of all data - has never been observed before and should not have occurred.
An investigation conducted by Digital Ocean has been inconclusive, but revealed no issues with the DB at the time of the incident or indications of ongoing issues.
Root cause is undetermined, but the incident appears to have been triggered by the backup operation.
What Next:
Without knowing why the DB failed in this way it is difficult to say what to do to keep it from happening again.
A future attempt to repeat this work will pull the backup from a DB replica instead of the Origin DB to (hopefully) minimize the impact if the issue should reoccur.
Members will be notified when this work has been scheduled.
Please take this as a reminder to check these forums or our status page at https://uptime.cosocial.info for information regarding planned or unplanned downtime.
Thank-you for your patience and support, and apologies for the inconvenience.