APIs - Order creation / AWB Registration, APIs - Tracking - AWS, and 9 other services are down
Resolved
Mar 13, 2026 at 02:02pm UTC
Root Cause Analysis (RCA): Application OOM Due to OpenTelemetry Java SDK Issue
Summary:
Our application instances experienced Out Of Memory (OOM) events that resulted in service instability instance restarts. The issue was traced to a known memory leak in the OpenTelemetry Java instrumentation described in OpenTelemetry Java Issue #8143.
The bug caused unbounded memory growth in the OpenTelemetry SDK during context propagation, eventually exhausting JVM heap memory under sustained traffic which lead to temporary degradation in service availability and increased latency during restarts.
Github link of the issue: https://github.com/open-telemetry/opentelemetry-java/issues/8143
Contributing Factors:
- Auto upgrade of minor version of packages introduced this bug into the code base.
- High request throughput magnified it
Resolution:
Mitigation steps taken:
- Upgraded OpenTelemetry Java SDK to a version containing the fix.
- Restarted affected services after deployment.
After the upgrade, heap usage stabilized and no further OOM events were observed.
Affected services
Tracking page - Basic
Updated
Mar 13, 2026 at 09:33am UTC
Tracking page - Basic recovered.
Affected services
Tracking page - Basic
Updated
Mar 13, 2026 at 09:33am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 09:32am UTC
Returns and Exchanges - Customer Portal recovered.
Affected services
Returns and Exchanges - Customer Portal
Updated
Mar 13, 2026 at 09:32am UTC
Tracking Page - personalised 1 and Tracking Page - personalised 2 recovered.
Affected services
Tracking Page - personalised 1
Tracking Page - personalised 2
Updated
Mar 13, 2026 at 09:32am UTC
APIs - Tracking - AWS recovered.
Affected services
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 09:31am UTC
Tracking Page - personalised 6 recovered.
Affected services
Tracking Page - personalised 6
Updated
Mar 13, 2026 at 09:31am UTC
Tracking Page - personalised 3 and Tracking Page - personalised 5 recovered.
Affected services
Tracking Page - personalised 3
Tracking Page - personalised 5
Updated
Mar 13, 2026 at 09:30am UTC
Tracking Page - personalised 4 recovered.
Affected services
Tracking Page - personalised 4
Updated
Mar 13, 2026 at 09:30am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 09:29am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 09:28am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 09:16am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 09:14am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 09:13am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 09:08am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 09:06am UTC
Tracking Page - personalised 3 and Tracking Page - personalised 5 went down.
Affected services
Tracking Page - personalised 3
Tracking Page - personalised 5
Updated
Mar 13, 2026 at 09:06am UTC
Tracking Page - personalised 1 went down.
Affected services
Tracking Page - personalised 1
Updated
Mar 13, 2026 at 09:05am UTC
APIs - Tracking - AWS went down.
Affected services
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 09:05am UTC
Tracking Page - personalised 1 recovered.
Affected services
Tracking Page - personalised 1
Updated
Mar 13, 2026 at 09:05am UTC
Tracking Page - personalised 4 went down.
Affected services
Tracking Page - personalised 4
Updated
Mar 13, 2026 at 09:05am UTC
APIs - Tracking - AWS recovered.
Affected services
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 09:04am UTC
Tracking Page - personalised 3, Tracking Page - personalised 4, and Tracking Page - personalised 5 recovered.
Affected services
Tracking Page - personalised 3
Tracking Page - personalised 4
Tracking Page - personalised 5
Updated
Mar 13, 2026 at 09:02am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 08:58am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 08:56am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 08:51am UTC
Dashboard - B2C [Backend] and Returns [Backend] went down.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 08:50am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 08:47am UTC
Tracking Page - personalised 1 went down.
Affected services
Tracking Page - personalised 1
Updated
Mar 13, 2026 at 08:47am UTC
Tracking Page - personalised 3 and Tracking Page - personalised 5 went down.
Affected services
Tracking Page - personalised 3
Tracking Page - personalised 5
Updated
Mar 13, 2026 at 08:46am UTC
Tracking Page - personalised 4 and Tracking Page - personalised 6 went down.
Affected services
Tracking Page - personalised 4
Tracking Page - personalised 6
Updated
Mar 13, 2026 at 08:45am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 08:45am UTC
Tracking Page - personalised 2 went down.
Affected services
Tracking Page - personalised 2
Updated
Mar 13, 2026 at 08:44am UTC
APIs - Order creation / AWB Registration and Dashboard - B2C [Backend] and Returns [Backend] went down.
Affected services
APIs - Order creation / AWB Registration
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 08:44am UTC
Tracking page - Basic went down.
Affected services
Tracking page - Basic
Updated
Mar 13, 2026 at 08:42am UTC
APIs - Tracking - AWS and Returns and Exchanges - Customer Portal went down.
Affected services
Returns and Exchanges - Customer Portal
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 08:41am UTC
Tracking Page - personalised 6 recovered.
Affected services
Tracking Page - personalised 6
Updated
Mar 13, 2026 at 08:39am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 08:38am UTC
Dashboard - B2C [Backend] and Returns [Backend] went down.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 08:27am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 08:23am UTC
Dashboard - B2C [Backend] and Returns [Backend] and Tracking Page - personalised 6 went down.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Tracking Page - personalised 6
Updated
Mar 13, 2026 at 08:18am UTC
Returns and Exchanges - Customer Portal recovered.
Affected services
Returns and Exchanges - Customer Portal
Updated
Mar 13, 2026 at 08:17am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 08:16am UTC
APIs - Tracking - AWS recovered.
Affected services
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 08:16am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 08:15am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 08:12am UTC
Dashboard - B2C [Backend] and Returns [Backend] went down.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 08:12am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 08:09am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 08:08am UTC
APIs - Tracking - AWS went down.
Affected services
APIs - Tracking - AWS
Created
Mar 13, 2026 at 08:08am UTC
Returns and Exchanges - Customer Portal went down.
Affected services
Returns and Exchanges - Customer Portal