Issue Summary Transaction (eligibility, alerts, estimates) processes were intermittently slower than usual or timed out for all interfaces.
Timeline First instances began at approximately 7:00 a.m. CT on Monday 6/1/2015 and ended at 11:00 a.m. CT on Monday 6/1/2015. Second instance began at approximately 1:45 p.m. CT on Monday 6/1/2015 and ended at 3:00 p.m. CT on Monday 6/1/2015.
Scope During the above mentioned times nearly 50% of all traffic was delayed beyond 60 seconds. Following these times, real-time processing was resumed but there was a backlog of transactions that were delayed in processing. This number may have been slightly higher or lower depending on how a specific client was routed.
Root Cause First of the month and first Monday of the month volume in addition to several large payers having processing issues revealed areas in our infrastructure that needed to be scaled out further. Additionally, the methods used to determine capacity planning have been adjusted with the data that was gathered during this time to allow more appropriate scaling of the infrastructure before an event like this occurs.
Resolution and Recovery Technical staff began evaluating and expanding those areas that were creating the backlog to handle the volume. Batch processing and non-critical processes were suspended to reduce volume and provide headroom for real-time processing. Worked with several large payers that were experiencing volume issues to identify root causes for their slowness.
Corrective and Preventative Measures Implemented additional performance tracking and active monitoring for these expanded areas. Evaluating opportunities identified during this slowness to improve processes for speedier recovery. Several development tickets have been created to address some internal applications that could handle volume in a more appropriate manner.
Passport/Experian Health apologizes for the inconvenience this caused to our customers and your patients.
