Webhook failures

Incident Report for Moodys

Postmortem

Root Cause Analysis - Webhook Processing Disruption (April 24, 2025)

‌

Impact
On April 24th, between 09:51 and 14:00 UTC, webhook processing was disrupted, leading to delays in message handling.

Root Cause
The incident was caused by a technical mismatch during a system update. This mismatch resulted in the creation of invalid messages, which blocked the processing of webhook queues.

Resolution
Our engineering team resolved the issue by correcting the database and manually fixing the affected messages.

Prevention Measures
We have identified improvements to prevent such disruptions in the future, including:

Upgrading systems to ensure reliable webhook delivery
Implementing measures to eliminate the possibility of data loss

Next Steps
We are committed to enhancing our systems and processes to prevent similar incidents in the future. Thank you for your understanding and continued support as we work to provide reliable service.

Posted May 08, 2025 - 16:10 BST

Resolved

This incident has been resolved.

Posted Apr 24, 2025 - 17:06 BST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Apr 24, 2025 - 14:46 BST

Update

We are continuing to work on a fix for this issue.

Posted Apr 24, 2025 - 13:41 BST

Identified

We have identified an issue in delivering webhooks.

We're investigating the cause and will provide an update as soon as possible.

Outstanding webhooks will be delivered once the issue is resolved.

Posted Apr 24, 2025 - 13:40 BST

This incident affected: Maxsight Environments (🇪🇺 EU - eu.maxsight.com).