Server Issues Causing Downtime

Incident Report for SportsEngine

Postmortem

We were alerted to a system issue starting at approximately 6:20pm CDT on 10/28 which impacted general website availability. After further investigation it was discovered that there was a disk issue on our master MySQL database. We were able to quickly restore general website availability. However we discovered that our MySQL replica databases were no longer receiving data updates. To avoid 20 minutes of data loss we were forced to put the platform into a read only mode which disabled login and online registration. In this state we were able to restore all of the data by launching new MySQL replica databases. At 10:20 CDT we re-enabled login and online registration and the issue was fully resolved. No data loss has occurred.

What happened was a result of a string of failures. We have two alerting systems in place that should have alerted us to the disk related issue prior to it occurring. Neither of these alerts occurred. We have an automatic task that runs nightly to ensure the database disk is in a top notch state, but this also failed to operate as expected. We will be undergoing a full audit of these systems to ensure that our alerts and the failing task work as expected to prevent this issue from occurring again. We are always striving to increase the uptime of the Sport Ngin platform and we sincerely apologize for this event. Thank you for your patience.

Posted Oct 28, 2013 - 23:07 CDT

Resolved

The issue has been resolved. Login and Registration are now enabled. Our databases are back and fully operational without the loss of data.

Posted Oct 28, 2013 - 22:20 CDT

Update

We have had a disk issue with one of our databases. We are working to ensure that no data is lost. We expect to have login and registration enabled in a few more hours. All content accessible to users that does not require login or is not registration related is fully operational. Thanks for your patience.

Posted Oct 28, 2013 - 21:11 CDT

Identified

The issue has been identified and we are working to fully resolve the issue. The Sport Ngin platform is still in a login disabled and registration disabled state.

Posted Oct 28, 2013 - 20:03 CDT

Investigating

We are investigating server issues that caused 10 minutes of downtime from 6:20 to 6:30 pm CST. While websites are back up we will be disabling login on the Sport Ngin platform while we investigate this issue further.

Posted Oct 28, 2013 - 18:55 CDT

This incident affected: SportsEngine HQ (Registration - programs and sign-ups, Website - front end CMS (Sitebuilder), Season Management - rostering, scheduling, location management, scoring, and stats, API - legacy integrations).