Elevated API errors

Incident Report for Moodys

Postmortem

Summary

Incident Period: March 10th, ~8:00 PM (GMT)

Issue: Some of our services became unavailable in the UAE, causing a significant number of incoming requests to fail.

Root Cause: Services were unable to connect to their database after the database server certificates were rotated.

Resolution: The certificate bundle on the services was updated.

Analysis

Services were unable to connect to the database after a certificate rotation, as the new certificate was issued by a certificate authority that was not trusted by our service.

Action

We updated our service's certificate bundle to include the new certificate authority, and are updating our internal processes to ensure we are better aware of future certificate rotations.

Posted Mar 13, 2025 - 16:44 GMT

Resolved

This incident has been resolved.

Posted Mar 11, 2025 - 08:53 GMT

Update

We are continuing to monitor for any further issues.

Posted Mar 11, 2025 - 08:53 GMT

Update

We are continuing to monitor for any further issues.

Posted Mar 11, 2025 - 08:36 GMT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Mar 10, 2025 - 21:45 GMT

Identified

The issue has been identified and we are continuing to work on a fix.

Posted Mar 10, 2025 - 21:42 GMT

Update

We are continuing to investigate this issue.

Posted Mar 10, 2025 - 21:38 GMT

Update

We are continuing to investigate this issue.

Posted Mar 10, 2025 - 21:22 GMT

Investigating

Maxsight is experiencing a higher number of API errors. This is also impacting the portal.

Our engineering team is currently working to identify the root cause and implement a solution.

Posted Mar 10, 2025 - 21:08 GMT

This incident affected: Maxsight Environments (🇦🇪 UAE - ae.maxsight.com).