Back to overview
Resolved

APIs - Order creation / AWB Registration, Tracking page - Basic, and 6 other services are down

Oct 30, 2026 at 3:53am UTC
Affected services
APIs - Order creation / AWB Registration
Tracking page - Basic
Tracking Page - personalised 1
Tracking Page - personalised 2
Tracking Page - personalised 3
Tracking Page - personalised 4
Tracking Page - personalised 6
Returns and Exchanges - Customer Portal

Resolved
Oct 31, 2026 at 2:54am UTC

Summary:
Customers experienced severe API latency and timeouts on www.clickpost.in specifically on create-order and serviceability API due to a sudden traffic spike at 12:15 IST.

Root cause:
A traffic surge on a quick-commerce serviceability end point, which sync calls carrier systems to check rider availability in real-time triggered request amplification (each request fans out to 100-200+ downstream calls), saturating the pgbouncer connection pool and triggering HTTP 429 (rate-limit) responses from third-party carrier APIs.

Resolution:
Implemented an API gateway at www.clickpost.in for rapid throttling of problematic endpoints, and increased pgbouncer pool_size.

Timeline:
2026-10-30, 12:15 IST: Sudden traffic spike on POST quickcom/serviceability (10,665 req/min)

12:15–12:50 IST: Pgbouncer pool (size=100) saturated; client waiting connections peaked at ~1,250

12:15 IST onwards: Third-party carrier APIs started returning HTTP 429 (Too Many Requests)

12:20–13:00 IST: API latency p99 climbed; thread parks reaching ~1152 ms per request

13:30 IST: Traffic subsided, system recovered

Post-incident:
API gateway deployed at www.clickpost.in; pgbouncer pool_size increased.

Symptoms:
Elevated API p99 latency across all endpoints

Thread pool exhaustion on application side.

Downstream pgbouncer backend connections saturated (sv_active hit 100/100)

Key Evidence:
Pgbouncer pool saturation (Grafana)

Client waiting connections (yellow line, peaked ~1,250)
Server active connections (blue line, hit 100 limit - pool_size cap)
Client active connections (green line, 400-600)

Time window: 12:15 - 13:30 IST

Pgbouncer SHOW POOLS confirmed sv_active = 100 with up to 1,250 clients queued.
SHOW DATABASES confirmed pool_size = 100 for the clickpost database.

One API request: 300–800 downstream calls.

At 10,665 req/min × ~500 downstream calls each = ~5.3 million downstream calls/min hitting pgbouncer + carrier APIs simultaneously.

Root Cause:
The quickcom serviceability endpoint has a built-in amplification factor of 100-800x due to its per-store looping design. During the 12:15 traffic spike:

Pgbouncer pool exhausted — pool_size = 100 was insufficient for the ~5M/min downstream query volume. Backend connections saturated, 1,250 clients queued waiting.

Third-party rate-limited — carrier APIs returned HTTP 429, forcing retries that compounded the load.

No edge throttling — there was no API-level rate limit to prevent a single endpoint from overwhelming shared infrastructure.

Resolution:

  1. API Gateway deployed at www.clickpost.in - centralized gateway now fronts all serviceability APIs, enabling rapid throttling or blocking of problematic endpoints without code deploys.

  2. Pgbouncer pool_size increased - from 100 to 200 to absorb burst load.

  3. Quickcom serviceability end point has been moved to a dedicated microservice, isolates its high fan-out load from the shared platform, preventing collateral impact on other APIs.

Updated
Oct 30, 2026 at 10:58am UTC

APIs - Order creation / AWB Registration recovered.

Updated
Oct 30, 2026 at 10:56am UTC

APIs - Order creation / AWB Registration went down.

Updated
Oct 30, 2026 at 10:52am UTC

APIs - Order creation / AWB Registration recovered.

Updated
Oct 30, 2026 at 10:51am UTC

APIs - Order creation / AWB Registration went down.

Updated
Oct 30, 2026 at 8:56am UTC

APIs - Order creation / AWB Registration recovered.

Updated
Oct 30, 2026 at 8:54am UTC

APIs - Order creation / AWB Registration went down.

Updated
Oct 30, 2026 at 7:20am UTC

APIs - Order creation / AWB Registration recovered.

Updated
Oct 30, 2026 at 7:00am UTC

APIs - Order creation / AWB Registration went down.

Updated
Oct 30, 2026 at 7:00am UTC

APIs - Order creation / AWB Registration recovered.

Updated
Oct 30, 2026 at 6:49am UTC

APIs - Order creation / AWB Registration went down.

Updated
Oct 30, 2026 at 4:56am UTC

Tracking Page - personalised 1 recovered.

Updated
Oct 30, 2026 at 4:54am UTC

Tracking Page - personalised 2, Tracking Page - personalised 3, and Tracking Page - personalised 4 recovered.

Updated
Oct 30, 2026 at 4:54am UTC

Tracking Page - personalised 3 went down.

Updated
Oct 30, 2026 at 4:53am UTC

Tracking Page - personalised 4 went down.

Updated
Oct 30, 2026 at 4:53am UTC

Tracking Page - personalised 1 went down.

Updated
Oct 30, 2026 at 4:52am UTC

Tracking Page - personalised 1 recovered.

Updated
Oct 30, 2026 at 4:51am UTC

Tracking Page - personalised 4 recovered.

Updated
Oct 30, 2026 at 4:51am UTC

Tracking Page - personalised 3 recovered.

Updated
Oct 30, 2026 at 4:49am UTC

Tracking Page - personalised 3 and Tracking Page - personalised 4 went down.

Updated
Oct 30, 2026 at 4:49am UTC

Tracking Page - personalised 3 recovered.

Updated
Oct 30, 2026 at 4:49am UTC

Tracking Page - personalised 4 recovered.

Updated
Oct 30, 2026 at 4:47am UTC

Tracking Page - personalised 2 and Tracking Page - personalised 3 went down.

Updated
Oct 30, 2026 at 4:47am UTC

Tracking Page - personalised 1 and Tracking Page - personalised 4 went down.

Updated
Oct 30, 2026 at 4:45am UTC

Tracking Page - personalised 1 and Tracking Page - personalised 2 recovered.

Updated
Oct 30, 2026 at 4:45am UTC

Tracking Page - personalised 3 recovered.

Updated
Oct 30, 2026 at 4:44am UTC

Tracking Page - personalised 4 recovered.

Updated
Oct 30, 2026 at 4:44am UTC

Tracking Page - personalised 6 recovered.

Updated
Oct 30, 2026 at 4:36am UTC

Tracking Page - personalised 3 went down.

Updated
Oct 30, 2026 at 4:35am UTC

Tracking Page - personalised 3 recovered.

Updated
Oct 30, 2026 at 4:35am UTC

Tracking Page - personalised 6 went down.

Updated
Oct 30, 2026 at 4:34am UTC

Tracking Page - personalised 6 recovered.

Updated
Oct 30, 2026 at 4:28am UTC

Tracking Page - personalised 6 went down.

Updated
Oct 30, 2026 at 4:27am UTC

Tracking Page - personalised 6 recovered.

Updated
Oct 30, 2026 at 4:12am UTC

Tracking page - Basic recovered.

Updated
Oct 30, 2026 at 4:10am UTC

Tracking Page - personalised 6 went down.

Updated
Oct 30, 2026 at 4:10am UTC

Returns and Exchanges - Customer Portal recovered.

Updated
Oct 30, 2026 at 4:07am UTC

Returns and Exchanges - Customer Portal went down and Tracking Page - personalised 6 recovered.

Updated
Oct 30, 2026 at 4:07am UTC

Tracking Page - personalised 6 went down.

Updated
Oct 30, 2026 at 4:06am UTC

Tracking page - Basic went down.

Updated
Oct 30, 2026 at 4:05am UTC

Tracking Page - personalised 4 went down.

Updated
Oct 30, 2026 at 4:05am UTC

Tracking Page - personalised 4 recovered.

Updated
Oct 30, 2026 at 3:59am UTC

Returns and Exchanges - Customer Portal recovered.

Updated
Oct 30, 2026 at 3:57am UTC

Tracking Page - personalised 4 went down.

Updated
Oct 30, 2026 at 3:56am UTC

Tracking Page - personalised 3 went down and Tracking Page - personalised 4 recovered.

Updated
Oct 30, 2026 at 3:55am UTC

Tracking Page - personalised 1, Tracking Page - personalised 2, and Tracking Page - personalised 4 went down.

Created
Oct 30, 2026 at 3:53am UTC

Returns and Exchanges - Customer Portal went down.