Root Cause: Modifications implemented to Systems Logging software due to locked files during log rotation.
Additionally, while POS Portal’s Platform Monitoring System generated the Alert when the outage began, the
Alert Type was not assigned to generate an ‘On Call Escalation’ for the outage to be immediately addressed. As
a result of this outage, Alerts have been reviewed and this Alert Type added to ‘On Call Escalation’.
Posted May 11, 2021 - 12:55 PDT
Resolved
POS Portal implemented a change the afternoon of Tuesday, May 4th to its Systems Logging software that monitors its Web Services applications. In the early hours of Thursday, May 6th, the Web Services platform became non-responsive to inbound requests due to a locked file during the log rotation process. While partners were able to attempt to process transactions through Web Services, they would not have received a response for the request. During the outage window, all requests made to POS Portal Web Services were unsuccessful, and did not result in the creation of any records in POS Portal’s system. This outage also impacted Portal Access.
Outage Window: Thursday, May 6th from 12:01AM – 3:27AM Pacific Time