APIs - Order creation / AWB Registration, Tracking page - Basic, and 6 other services are down
Resolved
Oct 31, 2026 at 2:54am UTC
Summary:
Customers experienced severe API latency and timeouts on www.clickpost.in specifically on create-order and serviceability API due to a sudden traffic spike at 12:15 IST.
Root cause:
A traffic surge on a quick-commerce serviceability end point, which sync calls carrier systems to check rider availability in real-time triggered request amplification (each request fans out to 100-200+ downstream calls), saturating the pgbouncer connection pool and triggering HTTP 429 (rate-limit) responses from third-party carrier APIs.
Resolution:
Implemented an API gateway at www.clickpost.in for rapid throttling of problematic endpoints, and increased pgbouncer pool_size.
Timeline:
2026-10-30, 12:15 IST: Sudden traffic spike on POST quickcom/serviceability (10,665 req/min)
12:15–12:50 IST: Pgbouncer pool (size=100) saturated; client waiting connections peaked at ~1,250
12:15 IST onwards: Third-party carrier APIs started returning HTTP 429 (Too Many Requests)
12:20–13:00 IST: API latency p99 climbed; thread parks reaching ~1152 ms per request
13:30 IST: Traffic subsided, system recovered
Post-incident:
API gateway deployed at www.clickpost.in; pgbouncer pool_size increased.
Symptoms:
Elevated API p99 latency across all endpoints
Thread pool exhaustion on application side.
Downstream pgbouncer backend connections saturated (sv_active hit 100/100)
Key Evidence:
Pgbouncer pool saturation (Grafana)
Client waiting connections (yellow line, peaked ~1,250)
Server active connections (blue line, hit 100 limit - pool_size cap)
Client active connections (green line, 400-600)
Time window: 12:15 - 13:30 IST
Pgbouncer SHOW POOLS confirmed sv_active = 100 with up to 1,250 clients queued.
SHOW DATABASES confirmed pool_size = 100 for the clickpost database.
One API request: 300–800 downstream calls.
At 10,665 req/min × ~500 downstream calls each = ~5.3 million downstream calls/min hitting pgbouncer + carrier APIs simultaneously.
Root Cause:
The quickcom serviceability endpoint has a built-in amplification factor of 100-800x due to its per-store looping design. During the 12:15 traffic spike:
Pgbouncer pool exhausted — pool_size = 100 was insufficient for the ~5M/min downstream query volume. Backend connections saturated, 1,250 clients queued waiting.
Third-party rate-limited — carrier APIs returned HTTP 429, forcing retries that compounded the load.
No edge throttling — there was no API-level rate limit to prevent a single endpoint from overwhelming shared infrastructure.
Resolution:
API Gateway deployed at www.clickpost.in - centralized gateway now fronts all serviceability APIs, enabling rapid throttling or blocking of problematic endpoints without code deploys.
Pgbouncer pool_size increased - from 100 to 200 to absorb burst load.
Quickcom serviceability end point has been moved to a dedicated microservice, isolates its high fan-out load from the shared platform, preventing collateral impact on other APIs.
Affected services
Updated
Oct 30, 2026 at 10:58am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
Updated
Oct 30, 2026 at 10:56am UTC
APIs - Order creation / AWB Registration went down.
Affected services
Updated
Oct 30, 2026 at 10:52am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
Updated
Oct 30, 2026 at 10:51am UTC
APIs - Order creation / AWB Registration went down.
Affected services
Updated
Oct 30, 2026 at 8:56am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
Updated
Oct 30, 2026 at 8:54am UTC
APIs - Order creation / AWB Registration went down.
Affected services
Updated
Oct 30, 2026 at 7:20am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
Updated
Oct 30, 2026 at 7:00am UTC
APIs - Order creation / AWB Registration went down.
Affected services
Updated
Oct 30, 2026 at 7:00am UTC
APIs - Order creation / AWB Registration recovered.
Affected services
Updated
Oct 30, 2026 at 6:49am UTC
APIs - Order creation / AWB Registration went down.
Affected services
Updated
Oct 30, 2026 at 4:56am UTC
Tracking Page - personalised 1 recovered.
Affected services
Updated
Oct 30, 2026 at 4:54am UTC
Tracking Page - personalised 2, Tracking Page - personalised 3, and Tracking Page - personalised 4 recovered.
Affected services
Updated
Oct 30, 2026 at 4:54am UTC
Tracking Page - personalised 3 went down.
Affected services
Updated
Oct 30, 2026 at 4:53am UTC
Tracking Page - personalised 4 went down.
Affected services
Updated
Oct 30, 2026 at 4:53am UTC
Tracking Page - personalised 1 went down.
Affected services
Updated
Oct 30, 2026 at 4:52am UTC
Tracking Page - personalised 1 recovered.
Affected services
Updated
Oct 30, 2026 at 4:51am UTC
Tracking Page - personalised 4 recovered.
Affected services
Updated
Oct 30, 2026 at 4:51am UTC
Tracking Page - personalised 3 recovered.
Affected services
Updated
Oct 30, 2026 at 4:49am UTC
Tracking Page - personalised 3 and Tracking Page - personalised 4 went down.
Affected services
Updated
Oct 30, 2026 at 4:49am UTC
Tracking Page - personalised 3 recovered.
Affected services
Updated
Oct 30, 2026 at 4:49am UTC
Tracking Page - personalised 4 recovered.
Affected services
Updated
Oct 30, 2026 at 4:47am UTC
Tracking Page - personalised 2 and Tracking Page - personalised 3 went down.
Affected services
Updated
Oct 30, 2026 at 4:47am UTC
Tracking Page - personalised 1 and Tracking Page - personalised 4 went down.
Affected services
Updated
Oct 30, 2026 at 4:45am UTC
Tracking Page - personalised 1 and Tracking Page - personalised 2 recovered.
Affected services
Updated
Oct 30, 2026 at 4:45am UTC
Tracking Page - personalised 3 recovered.
Affected services
Updated
Oct 30, 2026 at 4:44am UTC
Tracking Page - personalised 4 recovered.
Affected services
Updated
Oct 30, 2026 at 4:44am UTC
Tracking Page - personalised 6 recovered.
Affected services
Updated
Oct 30, 2026 at 4:36am UTC
Tracking Page - personalised 3 went down.
Affected services
Updated
Oct 30, 2026 at 4:35am UTC
Tracking Page - personalised 3 recovered.
Affected services
Updated
Oct 30, 2026 at 4:35am UTC
Tracking Page - personalised 6 went down.
Affected services
Updated
Oct 30, 2026 at 4:34am UTC
Tracking Page - personalised 6 recovered.
Affected services
Updated
Oct 30, 2026 at 4:28am UTC
Tracking Page - personalised 6 went down.
Affected services
Updated
Oct 30, 2026 at 4:27am UTC
Tracking Page - personalised 6 recovered.
Affected services
Updated
Oct 30, 2026 at 4:12am UTC
Tracking page - Basic recovered.
Affected services
Updated
Oct 30, 2026 at 4:10am UTC
Tracking Page - personalised 6 went down.
Affected services
Updated
Oct 30, 2026 at 4:10am UTC
Returns and Exchanges - Customer Portal recovered.
Affected services
Updated
Oct 30, 2026 at 4:07am UTC
Returns and Exchanges - Customer Portal went down and Tracking Page - personalised 6 recovered.
Affected services
Updated
Oct 30, 2026 at 4:07am UTC
Tracking Page - personalised 6 went down.
Affected services
Updated
Oct 30, 2026 at 4:06am UTC
Tracking page - Basic went down.
Affected services
Updated
Oct 30, 2026 at 4:05am UTC
Tracking Page - personalised 4 went down.
Affected services
Updated
Oct 30, 2026 at 4:05am UTC
Tracking Page - personalised 4 recovered.
Affected services
Updated
Oct 30, 2026 at 3:59am UTC
Returns and Exchanges - Customer Portal recovered.
Affected services
Updated
Oct 30, 2026 at 3:57am UTC
Tracking Page - personalised 4 went down.
Affected services
Updated
Oct 30, 2026 at 3:56am UTC
Tracking Page - personalised 3 went down and Tracking Page - personalised 4 recovered.
Affected services
Updated
Oct 30, 2026 at 3:55am UTC
Tracking Page - personalised 1, Tracking Page - personalised 2, and Tracking Page - personalised 4 went down.
Affected services
Created
Oct 30, 2026 at 3:53am UTC
Returns and Exchanges - Customer Portal went down.
Affected services