Incidents reported on status page for CodeLab DevOps https://codelabs-devops.betteruptime.com/ en https://codelabs-devops.betteruptime.com/incident/775090 Thu, 27 Nov 2025 07:21:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#a33d2d3ecd2f01d46c3c48cbde96c14a2adb298454e6cd13f2ab4e31fb9cc266 # Incident Post Mortem - Service Outage **Incident Date:** November 26, 2025 **Duration:** 16:40 CET - 20:16 CET (3 hours 36 minutes) ## Summary Following a backend deployment at 16:34 CET, our services failed to return online. Investigation revealed the root cause was insufficient memory resources on our backend virtual machine, not the deployment itself. ## Timeline - **16:34 CET** - Backend update deployed to production - **16:40 CET** - Services failed to come back online; incident investigation initiated - **17:30 CET** - Root cause identified as VM memory exhaustion - **19:25 CET** - Mitigation implemented to reduce RAM usage - **20:16 CET** - All services confirmed operational; monitoring continues ## Root Cause The virtual machine hosting our backend services exhausted available memory, preventing services from starting successfully after the deployment. ## Resolution We implemented measures to reduce the memory footprint of our services and will continue monitoring system resources closely to prevent recurrence. ## Next Steps We are reviewing our infrastructure capacity and monitoring thresholds to detect and prevent similar resource exhaustion issues in the future. https://codelabs-devops.betteruptime.com/incident/775090 Thu, 27 Nov 2025 07:21:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#a33d2d3ecd2f01d46c3c48cbde96c14a2adb298454e6cd13f2ab4e31fb9cc266 # Incident Post Mortem - Service Outage **Incident Date:** November 26, 2025 **Duration:** 16:40 CET - 20:16 CET (3 hours 36 minutes) ## Summary Following a backend deployment at 16:34 CET, our services failed to return online. Investigation revealed the root cause was insufficient memory resources on our backend virtual machine, not the deployment itself. ## Timeline - **16:34 CET** - Backend update deployed to production - **16:40 CET** - Services failed to come back online; incident investigation initiated - **17:30 CET** - Root cause identified as VM memory exhaustion - **19:25 CET** - Mitigation implemented to reduce RAM usage - **20:16 CET** - All services confirmed operational; monitoring continues ## Root Cause The virtual machine hosting our backend services exhausted available memory, preventing services from starting successfully after the deployment. ## Resolution We implemented measures to reduce the memory footprint of our services and will continue monitoring system resources closely to prevent recurrence. ## Next Steps We are reviewing our infrastructure capacity and monitoring thresholds to detect and prevent similar resource exhaustion issues in the future. https://codelabs-devops.betteruptime.com/incident/775090 Wed, 26 Nov 2025 19:16:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#17383c58cfe326291f91064f143e7547f60ee2ed13cd14c4b76932eb67f62237 All services have been restored and are operating normally. We will continue to monitor closely. We apologize for the disruption and thank you for your patience. A full post mortem will be published shortly. https://codelabs-devops.betteruptime.com/incident/775090 Wed, 26 Nov 2025 19:16:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#17383c58cfe326291f91064f143e7547f60ee2ed13cd14c4b76932eb67f62237 All services have been restored and are operating normally. We will continue to monitor closely. We apologize for the disruption and thank you for your patience. A full post mortem will be published shortly. https://codelabs-devops.betteruptime.com/incident/775090 Wed, 26 Nov 2025 18:31:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#9935329e7cc698ffd185fdef1616192b31c4a1e1770fb41c08f0b1385aca7220 We have implemented a fix to address the memory issues affecting our services. Our team is currently monitoring the situation closely to ensure stability is restored. Services should begin coming back online shortly. https://codelabs-devops.betteruptime.com/incident/775090 Wed, 26 Nov 2025 18:31:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#9935329e7cc698ffd185fdef1616192b31c4a1e1770fb41c08f0b1385aca7220 We have implemented a fix to address the memory issues affecting our services. Our team is currently monitoring the situation closely to ensure stability is restored. Services should begin coming back online shortly. https://codelabs-devops.betteruptime.com/incident/775090 Wed, 26 Nov 2025 16:31:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#75e68daa20fb9268845fe53cbda0c299ed14d1f063f3087a73e6d58e18fcafb6 We have identified the root cause of the outage. Our backend infrastructure is experiencing memory resource issues. Our team is working on implementing a fix. We apologize for the inconvenience and will continue to provide updates. https://codelabs-devops.betteruptime.com/incident/775090 Wed, 26 Nov 2025 16:31:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#75e68daa20fb9268845fe53cbda0c299ed14d1f063f3087a73e6d58e18fcafb6 We have identified the root cause of the outage. Our backend infrastructure is experiencing memory resource issues. Our team is working on implementing a fix. We apologize for the inconvenience and will continue to provide updates. https://codelabs-devops.betteruptime.com/incident/775090 Wed, 26 Nov 2025 15:41:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#e211dc81c45e7a876ae3f23d3522fed68bc58d0447d376d17fc3fd4905eb3486 We are currently experiencing an outage affecting our website and API. Most users are unable to access our services. Our team is actively investigating the issue and we will provide an update as soon as we have more information. https://codelabs-devops.betteruptime.com/incident/775090 Wed, 26 Nov 2025 15:41:00 -0000 https://codelabs-devops.betteruptime.com/incident/775090#e211dc81c45e7a876ae3f23d3522fed68bc58d0447d376d17fc3fd4905eb3486 We are currently experiencing an outage affecting our website and API. Most users are unable to access our services. Our team is actively investigating the issue and we will provide an update as soon as we have more information. https://codelabs-devops.betteruptime.com/incident/762231 Tue, 11 Nov 2025 13:55:54 -0000 https://codelabs-devops.betteruptime.com/incident/762231#0509401002a70af75d88cdd1173e3f60865084dc1ca710ee51828a15c98bd017 Maintenance completed https://codelabs-devops.betteruptime.com/incident/762231 Tue, 11 Nov 2025 13:55:54 -0000 https://codelabs-devops.betteruptime.com/incident/762231#0509401002a70af75d88cdd1173e3f60865084dc1ca710ee51828a15c98bd017 Maintenance completed https://codelabs-devops.betteruptime.com/incident/757663 Tue, 11 Nov 2025 11:45:00 -0000 https://codelabs-devops.betteruptime.com/incident/757663#1af363a927a9acd1e00ac4f97abd9b7c8dac6c48861f4c90eb26d3417f968be3 **Updated Nov 11 at 12:45pm CET** * **Status: Monitoring - Fix Deployed** * **Affected Service:** User Registration * **First Detected:** October 28, 2025 * **Last Updated:** November 11, 2025 ## Summary We have successfully deployed a fix for the registration issues. You can now create new accounts again! We're keeping a close eye on things to make sure everything continues running smoothly. ## What We Fixed The registration system wasn't properly checking that your password and password confirmation matched. This has now been corrected, and the registration process is working as intended. ## Current Status The fix is live and working. Our initial tests show: - ✓ You can successfully register with a new account - ✓ The system now properly checks that your passwords match - ✓ Registration is responding normally ## What's Next We'll continue monitoring the registration system over the next day or two to ensure everything remains stable. Once we're confident everything is running smoothly, we'll mark this incident as fully resolved. We sincerely apologize for the inconvenience this caused and appreciate your patience while we worked to resolve this issue. https://codelabs-devops.betteruptime.com/incident/762231 Tue, 11 Nov 2025 10:15:54 -0000 https://codelabs-devops.betteruptime.com/incident/762231#01897ed3818c65eb8d42b54e99976b51a3f09dbcdb135f8320f92acedd494945 **Scheduled Maintenance - Nov 11, 2025** * **Status: In Progress** * **Maintenance Window:** 11:15 AM - 2:55 PM CET (approx. 3h 40m) * **Affected Services:** User Registration, Authentication Systems * **Expected Impact:** Services may be slow or temporarily unavailable ## What We're Doing We're performing important maintenance to improve our registration and authentication systems. This includes: - Deploying fixes for password validation - Testing registration flow with real-world scenarios - Updating security configurations - Verifying database connections and user authentication ## What to Expect During this maintenance window: - **Registration** may be temporarily unavailable while we deploy updates - **Login** may experience brief interruptions during testing - **Existing sessions** should remain active, but you may need to log in again if disconnected - Some pages may load slower than usual ## What's Not Affected - Existing user sessions (in most cases) - Read-only access to public content - General browsing (outside of registration/login) ## Updates We'll post updates here if the maintenance takes longer than expected or if we finish early. Thank you for your patience while we make these improvements to keep your accounts secure! https://codelabs-devops.betteruptime.com/incident/762231 Tue, 11 Nov 2025 10:15:54 -0000 https://codelabs-devops.betteruptime.com/incident/762231#01897ed3818c65eb8d42b54e99976b51a3f09dbcdb135f8320f92acedd494945 **Scheduled Maintenance - Nov 11, 2025** * **Status: In Progress** * **Maintenance Window:** 11:15 AM - 2:55 PM CET (approx. 3h 40m) * **Affected Services:** User Registration, Authentication Systems * **Expected Impact:** Services may be slow or temporarily unavailable ## What We're Doing We're performing important maintenance to improve our registration and authentication systems. This includes: - Deploying fixes for password validation - Testing registration flow with real-world scenarios - Updating security configurations - Verifying database connections and user authentication ## What to Expect During this maintenance window: - **Registration** may be temporarily unavailable while we deploy updates - **Login** may experience brief interruptions during testing - **Existing sessions** should remain active, but you may need to log in again if disconnected - Some pages may load slower than usual ## What's Not Affected - Existing user sessions (in most cases) - Read-only access to public content - General browsing (outside of registration/login) ## Updates We'll post updates here if the maintenance takes longer than expected or if we finish early. Thank you for your patience while we make these improvements to keep your accounts secure! https://codelabs-devops.betteruptime.com/incident/762231 Tue, 11 Nov 2025 10:15:54 -0000 https://codelabs-devops.betteruptime.com/incident/762231#01897ed3818c65eb8d42b54e99976b51a3f09dbcdb135f8320f92acedd494945 **Scheduled Maintenance - Nov 11, 2025** * **Status: In Progress** * **Maintenance Window:** 11:15 AM - 2:55 PM CET (approx. 3h 40m) * **Affected Services:** User Registration, Authentication Systems * **Expected Impact:** Services may be slow or temporarily unavailable ## What We're Doing We're performing important maintenance to improve our registration and authentication systems. This includes: - Deploying fixes for password validation - Testing registration flow with real-world scenarios - Updating security configurations - Verifying database connections and user authentication ## What to Expect During this maintenance window: - **Registration** may be temporarily unavailable while we deploy updates - **Login** may experience brief interruptions during testing - **Existing sessions** should remain active, but you may need to log in again if disconnected - Some pages may load slower than usual ## What's Not Affected - Existing user sessions (in most cases) - Read-only access to public content - General browsing (outside of registration/login) ## Updates We'll post updates here if the maintenance takes longer than expected or if we finish early. Thank you for your patience while we make these improvements to keep your accounts secure! https://codelabs-devops.betteruptime.com/incident/762231 Tue, 11 Nov 2025 10:15:54 -0000 https://codelabs-devops.betteruptime.com/incident/762231#01897ed3818c65eb8d42b54e99976b51a3f09dbcdb135f8320f92acedd494945 **Scheduled Maintenance - Nov 11, 2025** * **Status: In Progress** * **Maintenance Window:** 11:15 AM - 2:55 PM CET (approx. 3h 40m) * **Affected Services:** User Registration, Authentication Systems * **Expected Impact:** Services may be slow or temporarily unavailable ## What We're Doing We're performing important maintenance to improve our registration and authentication systems. This includes: - Deploying fixes for password validation - Testing registration flow with real-world scenarios - Updating security configurations - Verifying database connections and user authentication ## What to Expect During this maintenance window: - **Registration** may be temporarily unavailable while we deploy updates - **Login** may experience brief interruptions during testing - **Existing sessions** should remain active, but you may need to log in again if disconnected - Some pages may load slower than usual ## What's Not Affected - Existing user sessions (in most cases) - Read-only access to public content - General browsing (outside of registration/login) ## Updates We'll post updates here if the maintenance takes longer than expected or if we finish early. Thank you for your patience while we make these improvements to keep your accounts secure! https://codelabs-devops.betteruptime.com/incident/757663 Tue, 04 Nov 2025 11:24:00 -0000 https://codelabs-devops.betteruptime.com/incident/757663#3f7549dd2d8d0e61e25e06859f103a9aaf01c6ee77e8c1c80f561fe4d6b39a6f - **Status:** Identified - Fix in Progress - **Affected Service:** User Registration API (/api/register) - **First Detected:** October 28, 2025 - **Last Updated:** November 4, 2025 ## Summary We have identified the root cause of the 403 Forbidden errors affecting our /api/register endpoint and are currently implementing a fix. Users are unable to complete new account registrations at this time. ## What We Know The /api/register endpoint has been returning 403 Forbidden errors since October 28, 2025 Root cause identified: Missing password confirmation validation in our registration endpoint The endpoint is not properly handling form-encoded registration requests ## Current Status Our engineering team has identified the specific code changes required to resolve this issue. We are currently: - Implementing password confirmation validation - Updating request parameter handling to support form-encoded data - Preparing the fix for deployment ## Next Steps - Complete code implementation and internal testing - Deploy fix to production environment - Verify registration functionality is fully restored - Conduct post-incident review We apologize for the inconvenience and will provide another update once the fix has been deployed. https://codelabs-devops.betteruptime.com/incident/757663 Tue, 04 Nov 2025 08:33:00 -0000 https://codelabs-devops.betteruptime.com/incident/757663#76ec959dfba0359b7ad48ea3879af60e137f5ba497ba66a524f2f03227e666c3 - **Status:** Investigating - **Affected Service:** User Registration API (/api/register) - **First Detected:** October 28, 2025 - **Last Updated:** November 4, 2025 ## Summary We are currently investigating issues with our /api/register endpoint that is returning 403 Forbidden errors. Our team has identified the scope of the issue and is actively working on a resolution. ## What We Know The /api/register endpoint is experiencing 403 Forbidden errors Multiple registration attempts have been affected since October 28, 2025 ## Current Status Our engineering team is actively investigating the root cause of the 403 errors on the registration endpoint. We are working to identify whether this is related to authentication, permissions, or configuration issues. ## What We're Doing - Analyzing server logs and error patterns - Reviewing recent changes to the registration service - Testing potential fixes in our staging environment We will provide updates as we learn more. Thank you for your patience. https://codelabs-devops.betteruptime.com/incident/754503 Thu, 30 Oct 2025 12:29:00 -0000 https://codelabs-devops.betteruptime.com/incident/754503#aaa51eb949450ee8faffd9de643b9995bc9a33cee6a5c484b55c905a1242ed82 We experienced a service crash and have recovered the application. We should be up and running and will continue to monitor the situation. https://codelabs-devops.betteruptime.com/incident/754503 Thu, 30 Oct 2025 12:29:00 -0000 https://codelabs-devops.betteruptime.com/incident/754503#aaa51eb949450ee8faffd9de643b9995bc9a33cee6a5c484b55c905a1242ed82 We experienced a service crash and have recovered the application. We should be up and running and will continue to monitor the situation. https://codelabs-devops.betteruptime.com/incident/754503 Thu, 30 Oct 2025 12:25:00 -0000 https://codelabs-devops.betteruptime.com/incident/754503#57424edc498a4add0cb5ff3345e44e5f7c24cf30675f121b5009c2665d15a787 We have identified the root cause of the issue. It will be resolved within 10-15 minutes. Thanks for your patience! https://codelabs-devops.betteruptime.com/incident/754503 Thu, 30 Oct 2025 12:25:00 -0000 https://codelabs-devops.betteruptime.com/incident/754503#57424edc498a4add0cb5ff3345e44e5f7c24cf30675f121b5009c2665d15a787 We have identified the root cause of the issue. It will be resolved within 10-15 minutes. Thanks for your patience! https://codelabs-devops.betteruptime.com/incident/754503 Thu, 30 Oct 2025 12:24:00 -0000 https://codelabs-devops.betteruptime.com/incident/754503#ad6c137c88a55a9cb2435b92db7694a89a924a4e2495af331d59ce2fb9da8408 We have registered the Website and API went down. We are currently investigating the cause. https://codelabs-devops.betteruptime.com/incident/754503 Thu, 30 Oct 2025 12:24:00 -0000 https://codelabs-devops.betteruptime.com/incident/754503#ad6c137c88a55a9cb2435b92db7694a89a924a4e2495af331d59ce2fb9da8408 We have registered the Website and API went down. We are currently investigating the cause. https://codelabs-devops.betteruptime.com/incident/728552 Mon, 22 Sep 2025 09:35:52 -0000 https://codelabs-devops.betteruptime.com/incident/728552#f8bfdb970df0379bd9d420644ba4860ea209ced2a84ce30c340f4a1b15165416 Frontend and API recovered. https://codelabs-devops.betteruptime.com/incident/728552 Mon, 22 Sep 2025 09:35:52 -0000 https://codelabs-devops.betteruptime.com/incident/728552#f8bfdb970df0379bd9d420644ba4860ea209ced2a84ce30c340f4a1b15165416 Frontend and API recovered. https://codelabs-devops.betteruptime.com/incident/728552 Mon, 22 Sep 2025 09:23:53 -0000 https://codelabs-devops.betteruptime.com/incident/728552#23284bfe34bf0aef4cf8ba5c5abe2b8273b3a52957263930b1087fcf8921273b Frontend went down. https://codelabs-devops.betteruptime.com/incident/728552 Mon, 22 Sep 2025 09:09:25 -0000 https://codelabs-devops.betteruptime.com/incident/728552#152deb18b2d9e897c4865fe9a22513c10ee4e1f41aa46dc8d8757bf0623008b9 API went down. https://codelabs-devops.betteruptime.com/incident/728526 Sun, 21 Sep 2025 00:06:00 -0000 https://codelabs-devops.betteruptime.com/incident/728526#0aa336236f1d250bf6ed101432e928a7acb7d2da6ec6abd8e1529dc9d2f4fff1 The app is back online, An endpoint seemed to have an unreachable route which caused the app to crash. We have corrected the issue and changed how we handle endpoints differently from view calls. https://codelabs-devops.betteruptime.com/incident/728526 Sun, 21 Sep 2025 00:02:00 -0000 https://codelabs-devops.betteruptime.com/incident/728526#ec56b6f2f5b39bc11c69be63c229d694443ca1e58f1d25d524a8eeb01c55dd87 We have identified the issue with an endpoint and bringing the app back online. https://codelabs-devops.betteruptime.com/incident/728526 Sun, 21 Sep 2025 00:01:00 -0000 https://codelabs-devops.betteruptime.com/incident/728526#30675c5a3ad03a21ee551463fd494d36a8c22b7b5440055007ae7ae0cfba3881 App seems to be offline. We will investigate.