Discourse Outage and VM failure


#1

Hi all,

The Discourse VM was caught in an unfortunate hypervisor failure and we’ve had to restore from backup. Sadly due to the timing, the most recent backup was from May 7th, just over a week ago, so data since then has been lost. Obviously I’m very sorry about this, and I’ll be posting a complete Root Cause Analysis to the main Announcements board once I’ve confirmed the site is back up and running.

Apologies again, and thanks for your patience.


#2

#3

The full RCA is now available here


#4

We are taking the site into read-only mode while we try to import the old posts. Stand by :slight_smile:


#5

Site is usable again, all posts and topics since the outage have been moved to higher database IDs so they wont conflict with the import and/or break threading. You may thus see a little weirdness with the posts made so far today - sorry about that.


#6

After investigation, I think I will be able to recover the missing posts, but it’s going to take me a little while to clean up the data. I should have it all restored by end of tomorrow - please hold off on resubmitting things until I’m done with that.


#7

I think I’m ready to attemp the import, but given the time I may as wait until the end of the day in US-East time. In a few hours I’ll put the site in read-only and see what I can do… :wink: