Full incident report

Wasabi status page

Degraded performance in US-EAST-1 and US-EAST-2 regions
An update has been posted
We are seeing a large amount of HTTP 500-level errors being returned to client requests due to this incident. Please check the status page regularly to receive the latest updates.
Time posted
May 30, 16:24 UTC

Ya think?

I'm inquiring with their salespeople if they have an option to mirror my files on a different server. I guess right now all my files are located on the East server that's down, but they've got other ones around the world. So, I wonder if they have a service where they can just switch it over. I know it sucks for you guys not being able to watch any videos today, and I apologize for that. So, I'm looking into options. I've been with Wasabi for many years, and I've never had a major problem like this before. They've been down for a few minutes here and there, but never for hours. I, of course, have of all my files backed up locally but that doesn't do me any good on the website.

Since this seems to be only an issue affecting their East Coast Data Centers, I've created a replication bucket in their West Coast Data Center. As soon as all the files (200 GB worth) are done copying, I will point the website at the new bucket. For those of you who are unfamiliar, a bucket is a term for a folder, basically, in a nutshell, on a file storage server.

Right now, I'm just waiting for it to replicate, and I'm going to keep them replicated in case this happens in the future. Unless Wasabi goes through some kind of a system-wide outage, which is very rare, we should be good to go. Apparently, this problem is only affecting one Data Center, and they've got many data centers.

Haha. Never fails. Looks like the East data center is coming back online and videos are starting to work JUST as I'm getting ready to implement my backup bucket change. LOL.

An update has been posted
Recovery operation is currently underway to bring back the impacted systems. This will take between 6 to 12 hours to complete. We will continue to update here as progress is made.
Time posted
May 30, 19:29 UTC

Looks as though we're back up and running. I did spot checks in a few videos and downloads, and from what I can tell, everything's back to normal. However, because of today, I now have a backup bucket that I can quickly redirect to in the event that something like this happens again. Fingers crossed.

An update has been posted
We are continuing the restoration of the impacted components. A large number of servers have been rebooted & restored and we are working on the remaining servers in both regions. We anticipate the full process to complete between 4-8 hours. We will continue to update our status page.
Time posted
May 31, 01:09 UTC

Incident status: Monitoring
Systems have been restored to operational status.
We continue to monitor these services.
If you experience any issue please reach out to our support team.

Richard Hi, this may not be related to the above but the recently added Text Box Hints (https://599cd.com/blog/display-article.asp?ID=3201) video stops playing at 1 min 43 secs.

Hey David, it's playing fine for me, and no one else has complained. Sometimes videos do get corrupted over time; I don't know why. But in this particular case, the video seems okay. If a video is actually corrupted it won't play for me either. What likely happened was when the server was having issues, your browser downloaded only a partial copy of the video file. So, clear your browser cache and that should fix the problem for you. If not, let me know.

Thanks Richard. I left things for a couple of hours and it's playing fine now. I haven't had a gremlin for a while! :o)