Delays in PDF Generation, Emails

Incident Report for TravelBank

Postmortem

The delays with expense export, PDF generation, and emails is an issue that is occurring intermittently due to an issue with our reporting database.

These processes are all implemented via a job/worker system. Due to the issue with our reporting database, sometimes these jobs are getting locked, which is causing all subsequent jobs to be delayed. Resolution at the moment is a manual process that involves making updates to our reporting database.

We are actively working on a resolution to prevent this from happening. We hope to have a resolution as soon as possible, but it may take more than a day. In addition, we are working on improving our internal and external communications to better respond to these incidents when they do occur.

We apologize for the recent and ongoing service disruptions and understand that when the system is not working as expected, our customers are not getting the value and quality they deserve. We are working with urgency to get this resolved.

-TravelBank Engineering, Adam Hong

Posted Mar 01, 2024 - 08:53 PST

Resolved

This incident has been resolved.

Posted Feb 27, 2024 - 16:33 PST

Update

Delays in emails and PDF generation should now be resolved. This incident was related to our reporting incident.

Posted Feb 27, 2024 - 11:19 PST

Update

There will be a temporarily delay in emails and PDF generation as we try to restore our reporting database.

Posted Feb 27, 2024 - 10:40 PST

Monitoring

From approximately 5am ET until 1pm ET on Tuesday, February 27th, certain processes were delayed including PDF generation and emails due to an infrastructure event. This has now been resolved. We are working on changes to the system that will reduce the impact to critical operations such as emails when an event like this occurs.

Posted Feb 27, 2024 - 10:01 PST

This incident affected: Core (Login, API).