[5:00pm CT]
Post Mortem
Issue Summary Transaction: eCare NEXT (eligibility, alerts, estimates) processes were intermittently slower than usual or timed out for all interfaces.
Timeline: Slowness began to affect customers at approximately 9:30 a.m. CT on Friday 6/19/2015 and ended at 11:30 a.m. CT on Friday 6/19/2015.
Scope: All eCare NEXT and IntelliSource customers impacted.
Root Cause: A change moved to production to fix an issue with compression inadvertently created a memory issue in our infrastructure that caused services and transactions to backlog.
Resolution and Recovery: The change was moved out of production
Corrective and Preventative Measures: Although this change was thoroughly tested on our testing platform, we could not replicate this issue. We will continue to utilize more extensive testing scenarios and improve our ability to roll back changes more easily.
[12:00pm CT] Please be aware that RQA/Alerts may be slightly delayed as we process the backlog. We anticipate all processing to be back to normal by 1pm CT.
[11:00am CT] We have rolled back changes that created the slowness and are working on clearing the backlog. Transaction processing should be back to normal before 11:30am. We apologize for the disruption.
[9:00am CT] We are researching an issue that is causing some customers to experience processing slowness. Our application and system teams are working on a resolution. As soon as we receive an estimated fix time, we will communicate. Thanks for your patience!
