Populi unavailable for a large subset of users
Incident Report for Populi
Resolved
From roughly 4:20am - 4:50am Pacific time a large subset of Populi users were unable to access the service. This was because a weekly database consistency check started running at roughly 3am and eventually locked up a critical database server, leading to widespread system failure. We eventually managed to restart the database and restore service, and we'll be taking a close look at that check today to prevent it causing issues in the future!
Posted Jul 03, 2023 - 04:30 PDT