Due to some disk issues during this night backups, some of our MongoDB databases crashed. For the same reason, the automatic restart procedure did not work. This sudden crash confused MongoDB causing some WiredTiger file corruptions and prevented the impacted databases to be restarted.
Our infrastructure provider did a snapshot of the disk yesterday at 8PM. Restoring this snapshot took a couple of hours. Then we copied the WiredTiger files from the snapshot and started a MongoDB repair process with the most recent MongoDB data.
We ended up restoring all the data, without any data loss for your applications.
If any of you still has trouble with a MongoDB database, reach our support team as soon as possible at email@example.com or via the embedded chat on the dashboard.
A more detailed post-mortem will be released soon.