Issue Summary

On February 9, 2020 all API requests to POS Portal API Web services began receiving error response message "The provided Timestamp header is outside of the acceptable bounds. Please retry using the current time.”. This impacted API requests from Portal Cloud, Portal Access, Direct Merchant eCommerce, and Re-seller eCommerce.

Timeline (all times Pacific Standard Time)

• 11:35 AM: Outage reported
• 12:02 PM: Notified IT On-Call support
• 12:11 PM: Issue was identified
• 12:15 PM: Issue was resolved
• 12:17 PM: API service restored

Root Cause

The issue was identified that one of the API back-end system clock was out of synchronization with official time sources and caused a time drift of more than 3 minutes. This difference in time between client and server caused API authentication errors and rejected all API requests.

Resolution and recovery

System clocks were manually synchronized with official time sources

Corrective and Preventative Measures

· Automatic clock synchronization tool installed to prevent further time drifts on the identified servers. A full review of all Production servers will be completed and Hardening Policies will be reviewed and updated as needed.

· Monitoring and alerting of clock synchronization and drift is being implemented.