APIs - Order creation / AWB Registration, APIs - Tracking - AWS, and 9 other services are down
Resolved
Mar 13, 2026 at 2:02pm UTC
Root Cause Analysis (RCA): Application OOM Due to OpenTelemetry Java SDK Issue
Summary:
Our application instances experienced Out Of Memory (OOM) events that resulted in service instability instance restarts. The issue was traced to a known memory leak in the OpenTelemetry Java instrumentation described in OpenTelemetry Java Issue #8143.
The bug caused unbounded memory growth in the OpenTelemetry SDK during context propagation, eventually exhausting JVM heap memory under sustained traffic which lead to temporary degradation in service availability and increased latency during restarts.
Github link of the issue: https://github.com/open-telemetry/opentelemetry-java/issues/8143
Contributing Factors:
- Auto upgrade of minor version of packages introduced this bug into the code base.
- High request throughput magnified it
Resolution:
Mitigation steps taken:
- Upgraded OpenTelemetry Java SDK to a version containing the fix.
- Restarted affected services after deployment.
After the upgrade, heap usage stabilized and no further OOM events were observed.
Affected services
Tracking page - Basic
Updated
Mar 13, 2026 at 9:33am UTC
Tracking page - Basic recovered.
Affected services
Tracking page - Basic
Updated
Mar 13, 2026 at 9:33am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 9:32am UTC
Returns and Exchanges - Customer Portal recovered.
Affected services
Returns and Exchanges - Customer Portal
Updated
Mar 13, 2026 at 9:32am UTC
Tracking Page - personalised 1 and Tracking Page - personalised 2 recovered.
Affected services
Tracking Page - personalised 1
Tracking Page - personalised 2
Updated
Mar 13, 2026 at 9:32am UTC
APIs - Tracking - AWS recovered.
Affected services
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 9:31am UTC
Tracking Page - personalised 6 recovered.
Affected services
Tracking Page - personalised 6
Updated
Mar 13, 2026 at 9:31am UTC
Tracking Page - personalised 3 and Tracking Page - personalised 5 recovered.
Affected services
Tracking Page - personalised 3
Tracking Page - personalised 5
Updated
Mar 13, 2026 at 9:30am UTC
Tracking Page - personalised 4 recovered.
Affected services
Tracking Page - personalised 4
Updated
Mar 13, 2026 at 9:30am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 9:29am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 9:28am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 9:16am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 9:14am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 9:13am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 9:08am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 9:06am UTC
Tracking Page - personalised 3 and Tracking Page - personalised 5 went down.
Affected services
Tracking Page - personalised 3
Tracking Page - personalised 5
Updated
Mar 13, 2026 at 9:06am UTC
Tracking Page - personalised 1 went down.
Affected services
Tracking Page - personalised 1
Updated
Mar 13, 2026 at 9:05am UTC
APIs - Tracking - AWS went down.
Affected services
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 9:05am UTC
Tracking Page - personalised 1 recovered.
Affected services
Tracking Page - personalised 1
Updated
Mar 13, 2026 at 9:05am UTC
Tracking Page - personalised 4 went down.
Affected services
Tracking Page - personalised 4
Updated
Mar 13, 2026 at 9:05am UTC
APIs - Tracking - AWS recovered.
Affected services
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 9:04am UTC
Tracking Page - personalised 3, Tracking Page - personalised 4, and Tracking Page - personalised 5 recovered.
Affected services
Tracking Page - personalised 3
Tracking Page - personalised 4
Tracking Page - personalised 5
Updated
Mar 13, 2026 at 9:02am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 8:58am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 8:56am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 8:51am UTC
Dashboard - B2C [Backend] and Returns [Backend] went down.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 8:50am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 8:47am UTC
Tracking Page - personalised 1 went down.
Affected services
Tracking Page - personalised 1
Updated
Mar 13, 2026 at 8:47am UTC
Tracking Page - personalised 3 and Tracking Page - personalised 5 went down.
Affected services
Tracking Page - personalised 3
Tracking Page - personalised 5
Updated
Mar 13, 2026 at 8:46am UTC
Tracking Page - personalised 4 and Tracking Page - personalised 6 went down.
Affected services
Tracking Page - personalised 4
Tracking Page - personalised 6
Updated
Mar 13, 2026 at 8:45am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 8:45am UTC
Tracking Page - personalised 2 went down.
Affected services
Tracking Page - personalised 2
Updated
Mar 13, 2026 at 8:44am UTC
APIs - Order creation / AWB Registration and Dashboard - B2C [Backend] and Returns [Backend] went down.
Affected services
APIs - Order creation / AWB Registration
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 8:44am UTC
Tracking page - Basic went down.
Affected services
Tracking page - Basic
Updated
Mar 13, 2026 at 8:42am UTC
APIs - Tracking - AWS and Returns and Exchanges - Customer Portal went down.
Affected services
APIs - Tracking - AWS
Returns and Exchanges - Customer Portal
Updated
Mar 13, 2026 at 8:41am UTC
Tracking Page - personalised 6 recovered.
Affected services
Tracking Page - personalised 6
Updated
Mar 13, 2026 at 8:39am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 8:38am UTC
Dashboard - B2C [Backend] and Returns [Backend] went down.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 8:27am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 8:23am UTC
Dashboard - B2C [Backend] and Returns [Backend] and Tracking Page - personalised 6 went down.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Tracking Page - personalised 6
Updated
Mar 13, 2026 at 8:18am UTC
Returns and Exchanges - Customer Portal recovered.
Affected services
Returns and Exchanges - Customer Portal
Updated
Mar 13, 2026 at 8:17am UTC
Dashboard - B2C [Backend] and Returns [Backend] recovered.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 8:16am UTC
APIs - Tracking - AWS recovered.
Affected services
APIs - Tracking - AWS
Updated
Mar 13, 2026 at 8:16am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 8:15am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 8:12am UTC
Dashboard - B2C [Backend] and Returns [Backend] went down.
Affected services
Dashboard - B2C [Backend] and Returns [Backend]
Updated
Mar 13, 2026 at 8:12am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 8:09am UTC
APIs - Order creation / AWB Registration went down.
Affected services
APIs - Order creation / AWB Registration
Updated
Mar 13, 2026 at 8:08am UTC
APIs - Tracking - AWS went down.
Affected services
APIs - Tracking - AWS
Created
Mar 13, 2026 at 8:08am UTC
Returns and Exchanges - Customer Portal went down.
Affected services
Returns and Exchanges - Customer Portal