On October 19, between approximately 11:05 and 12:00 CEST, the Voyado Engage APIs experienced degraded performance and some API requests failed, this was due to an infrastructure issue affecting our cloud provider. The issue caused increased response times across our API services, which were further amplified by an internal traffic management mechanism. We identified the contributing factors and took action to restore normal performance.
During the incident, customers experienced slower response times when making requests to the Voyado Engage APIs. Some customers also experienced failed requests, with temporary errors (503). Performance was back to normal at about 12:00 CEST.
Other platform functionality was confirmed not to be affected.
The incident was initially triggered by an infrastructure issue at our cloud provider, which caused reduced performance across multiple services simultaneously. This triggered an internal traffic management mechanism designed to balance load and isolate unusually slow API traffic to protect overall platform performance. The mechanism did not manage the situation as intended which caused delays and some failed API requests for affected customers.
As part of our mitigation efforts, we increased the capacity available to handle the redirected API traffic. Alongside the recovery of the underlying cloud infrastructure, this helped restore normal API performance by approximately 12:00 CEST on October 19.
We continued monitoring the platform to ensure stability before formally closing the incident on October 20.
To reduce the risk of similar incidents, we are reviewing and taking appopriate actions of the following:
We remain committed to improving the reliability and resilience of Voyado Engage and appreciate your patience and understanding. We apologize for the disruption this caused.