Issue Summary

On February 13, 2020 all API requests to POS Portal API Web services began receiving error response message "The provided Timestamp header is outside of the acceptable bounds. Please retry using the current time.”. This impacted API requests from Portal Cloud, Portal Access, Direct Merchant eCommerce, and Re-seller eCommerce.

Timeline (all times Pacific Standard Time)

• 2/13/2020 - 22:42 AM: Outage started
• 2/14/2020 – 20:53 PM: Issue resolved

Root Cause

The issue was identified that one of the API back-end system clock was out of synchronization with official time sources and caused a time drift 182 seconds. This difference in time between client and server caused API authentication errors and rejected all API requests.

Resolution and recovery

System clocks were manually synchronized with official time sources.

Corrective and Preventative Measures

• Implemented a control that periodically checks for time drift and adjusts accordingly to official time sources.
• A review of all system configurations will be completed. Additionally, a review of all Production environments will be completed to ensure all systems are in compliance standard and monitoring.