Summary
On 2 November 2026, from 16:08 to 17:02 IST (54 minutes), requests to our v3 order creation API failed with server errors (HTTP 5xx) for most carriers. Orders for the 91 carriers handled by our newer order creation service were not affected. The failure was caused by legacy code that wrote to a deprecated logging table, which could no longer store record IDs once they exceeded the maximum value of a 32-bit integer. Our monitoring detected the elevated error rate, and our engineering team disabled the failing component and restored service at 17:02 IST.

What Happened
Background: the deprecated logging table. Our order creation runs on two services: a newer service written in Kotlin, and an older Python-based service that handles v3 order creation for all other carriers. After an order is created, our post-processing service performs follow-up processing. Previously, the post-processing service also wrote a log entry to a dedicated logging table, recording the ID of a related record. That logging table was deprecated in one of our February 2026 releases.

What was left behind. When the logging table was deprecated, the code that wrote to it was removed from the path used by the Kotlin service, but not from the path used by the Python-based service. The column in the logging table that stored the related record's ID used a 32-bit signed integer type, which holds values up to 2,147,483,647. The related records themselves use IDs without that limit, so their IDs kept growing past the capacity of the logging column. The Kotlin service was unaffected because it no longer called the legacy logging code at all.

Steps that were required but missed.
1. Removal of the legacy logging write from every order creation path when the table was deprecated in February 2026. It was removed from the Kotlin path only.
2. Either widening the logging column to a 64-bit integer or retiring the table entirely, before related record IDs reached 2,147,483,647 at approximately 16:08 IST on 2 November 2026.
3. Capacity monitoring of the logging column. Our integer capacity checks covered only primary key and foreign key columns. This column stored IDs from another table but was not declared as a foreign key, so it was never checked and no alert was raised as it approached its limit.

Failure chain.
1. At approximately 16:08 IST, the ID of a newly created related record exceeded 2,147,483,647.
2. The database rejected the write to the logging table, because the value no longer fit in the 32-bit column.
3. The logging write ran synchronously, within the same request as order creation, so the error caused the whole order creation request to fail.
4. Customers creating orders through the v3 API for carriers handled by the Python-based service received HTTP 5xx errors.

Detection and Response
Our automated monitoring detected the issue within a minute, raising an alert for elevated HTTP 5xx error rates on order creation, and the on-call engineering team responsible for order creation was alerted immediately. By comparing our two order creation services, the team saw failures on the Python-based service only. That narrowed the search to legacy code in the post-processing service writing to the deprecated logging table. The team disabled that write and restored order creation at 17:02 IST.

Root Cause
Legacy code in our post-processing service continued to write to a deprecated logging table that stored related record IDs in a 32-bit integer column. When those IDs exceeded 2,147,483,647, the database rejected the writes, and because the write was synchronous, each rejection caused the corresponding v3 order creation request to fail.

Contributing Factors
1. Incomplete decommissioning. The logging table was deprecated, but the code writing to it was removed from only one of the two order creation paths. There was no check confirming that all writes to the table had stopped.
2. Gaps in capacity monitoring. Our integer capacity checks covered primary key and foreign key columns only. Columns that store IDs from other tables without a declared foreign key, or that hold user-supplied integers, were not monitored.
3. Logging in the critical path. The logging write ran synchronously as part of order creation, so a failure in a non-essential logging step blocked the order itself.

Resolution
We disabled the legacy logging write in the post-processing service, which immediately stopped the failing writes, and order creation was fully restored at 17:02 IST. We then deployed a code change that permanently removes the legacy logging write from the Python-based order creation path. Restoration was confirmed by our error monitoring system, with all related alarms resolved.

Long-Term Corrective Actions
Every contributing factor is addressed by at least one action below, so the same failure cannot recur through any of the three gaps:
1. Unmonitored ID columns (actions 1, 2 and 3): every integer column that stores IDs or user-supplied values is found, monitored, and where possible widened to 64 bits so the limit no longer applies.
2. Logging in the critical path (action 4): a logging failure can no longer fail an order creation request.
3. Incomplete decommissioning (action 5): a table cannot be marked deprecated while any code still writes to it.