Incident Post Mortem - Service Outage

Incident Date: November 26, 2025

Duration: 16:40 CET - 20:16 CET (3 hours 36 minutes)

Summary

Following a backend deployment at 16:34 CET, our services failed to return online. Investigation revealed the root cause was insufficient memory resources on our backend virtual machine, not the deployment itself.

Timeline

• 16:34 CET - Backend update deployed to production
• 16:40 CET - Services failed to come back online; incident investigation initiated
• 17:30 CET - Root cause identified as VM memory exhaustion
• 19:25 CET - Mitigation implemented to reduce RAM usage
• 20:16 CET - All services confirmed operational; monitoring continues

Root Cause

The virtual machine hosting our backend services exhausted available memory, preventing services from starting successfully after the deployment.

Resolution

We implemented measures to reduce the memory footprint of our services and will continue monitoring system resources closely to prevent recurrence.

Next Steps

We are reviewing our infrastructure capacity and monitoring thresholds to detect and prevent similar resource exhaustion issues in the future.

All services have been restored and are operating normally. We will continue to monitor closely. We apologize for the disruption and thank you for your patience. A full post mortem will be published shortly.

We have implemented a fix to address the memory issues affecting our services. Our team is currently monitoring the situation closely to ensure stability is restored. Services should begin coming back online shortly.

We have identified the root cause of the outage. Our backend infrastructure is experiencing memory resource issues. Our team is working on implementing a fix. We apologize for the inconvenience and will continue to provide updates.

We are currently experiencing an outage affecting our website and API. Most users are unable to access our services. Our team is actively investigating the issue and we will provide an update as soon as we have more information.