System Issue: April 11/2024
Subject: System Down
Our engineers are currently looking into issues with system access and are working to restore functionality. Updates to follow as quickly as possible.
-- RESOLVED -- 5/11/2024 6:50am PT
System dashboard has been restored and the platform is now accessible. Kiosks are also functioning again.
You will experience some slowness as our team continues to monitor and optimize the platform and ensure everything is working as intended.
Our engineers will continue to investigate this morning's incident and will provide a more detailed explanation which we will send to users on Monday.
Thank you for your patience and understanding and we apologize for any inconvenience this may have caused.
-- RESOLUTION UPDATE -- 5/13/2024 9:15am PT
On Saturday, May 11th at approximately 1am PT (4am ET), a hard drive within a RAID (Redundant Array of Independent Disks) storage array on one of our secure colocation-hosted servers experienced a hardware malfunction. This malfunction subsequently caused the RAID array controller software to crash, preventing the array from switching to another active storage drive node, resulting in a drive error on the server.
At 5:15am PT, first reports of the software outage were received from users and our support team was engaged and communicating with those affected.. System engineers were online at 5:45am PT and all services were fully restored and caught up by 6:45am PT with minimal impact to users.
Engineers have taken steps to move some processes from the impacted RAID array to backup servers while they work to replace the malfunctioned drive. No data has been lost as a result of the malfunctioned drive and all system processes are operational at this time.
We apologize for any disruption in service this may have caused to you and your customers.