Users connecting to the server began experiencing intermittent errors at 2025-11-13T07:19:41Z.
2025-11-13T11:05:00Z
- The server has been taken offline to address the database storage issues that caused the failure.
- Repair is under way.
- Expected resolution within 30 minutes.
- Updates to follow.
2025-11-13T11:50:00Z
- An issue has been encountered with Digital Ocean’s database system which is preventing repair of the issue at this time
- I am continuing to work to address the issue as quickly as possible
- Updates to follow
2025-11-13T12:00:00Z
- The issue has been resolved and the system is back online
- Our timelines are rebuilding with messages that were not delivered during the downtime. Everything should be caught up in <30 minutes.
2025-11-13T12:30:00Z
- All backlogged queues have cleared and the incident has been resolved at this time
What Happened?
This is partly a continuation of the incident that occurred on 2025-11-07, when an attempt to perform a database space increase failed due to errors that occurred when making a database backup.
That maintenance failed and was due to be repeated this morning.
Unfortunately, the database became critically low on space sooner than expected, and entered read-only mode at 2025-11-13T07:19:00Z.
An additional delay was encountered when Digital Ocean’s system failed to acknowledge the initial request to add more space to the DB cluster.
Subsequent attempts were successful and this issue is under investigation by Digital Ocean.
What Next?
Digital Ocean has recently provided the option to have the database cluster automatically re-size as space gets low to prevent issues of this sort.
This feature is now enabled on our cluster, which will incrementally increase the size of the database by 10 GB as space gets low.
Space constraints of this sort will not be an issue when we have completed the migration to a new hosting provider, which is anticipated by the end of the year.
My apologies for the inconvenience.
Thank-you for your patience and support.