Services have been restored back to normal state.

Summary

A scheduled database maintenance activity led to a temporary failure in order data persistence, which cascaded into failures on order-related API requests. Our monitoring detected the issue promptly, and engineering restored full service shortly after. The incident started at 05:13 PM IST and was fully resolved by 06:26 PM IST.

What Happened

From 12th to 19th September, the Clickpost team performed planned maintenance to migrate the primary key columns and ID sequences of our core tables from integer to bigint. The migration involved updating our data pipelines to handle the schema changes and updating the sequences across all involved databases. The data pipeline maintenance was completed on 16th September at 2:30 AM IST, and the overall maintenance was completed on 19th September at 11:30 PM IST. The activity was planned as a zero-downtime maintenance.

Our Order Creation Service, which is responsible for AWB creation and registration, uses jOOQ for database access. jOOQ generates Java classes from the database schema, and these classes enforce column data types strictly. As part of the migration, two steps were required for this service before 18th September:

1. Application patch: regenerate the service's jOOQ classes to match the new bigint data types in the database schema, and redeploy the application.

2. Configuration check: add test cases to validate that the service correctly handles IDs above the 32-bit integer limit (2,147,483,647).

As part of the migration, the Order Creation and shipment registration service required a compatibility update to align with the new bigint schema, along with validation for IDs beyond the 32-bit integer range. This specific scenario, new IDs crossing the 32-bit boundary, fell outside the validation scope defined for this migration phase. When IDs began exceeding this range on 20th September, the Order Creation Service was unable to process them during order writes, which cascaded into the API failures observed during that window.

Detection and Response

Automated monitoring detected elevated error rates and persistence failures promptly. On-call engineering was paged immediately and traced the issue to the missed jOOQ class regeneration and redeployment. Because remediation required regenerating the classes, patching, and verifying the service manually, the rollout took longer than an automated rollback would have.

Root Cause

The Order Creation Service's jOOQ classes still mapped the primary key columns as 32-bit integers after the database migration to bigint. Once IDs exceeded the 32-bit limit, the service rejected the values during order writes.

Contributing factor: a gap in SOP coverage. The migration SOP did not include regenerating jOOQ classes, redeploying jOOQ-based services, or testing against IDs beyond the 32-bit limit, so the mismatch reached production.

Resolution

The jOOQ classes were regenerated against the updated schema, and the patched Order Creation Service was redeployed. API requests resumed processing successfully, and monitoring confirmed full service restoration.

Long-Term Corrective Actions

1. SOP update: We have updated our infrastructure maintenance SOPs to close this gap. Any core data type or schema change now requires mandatory sign-off on application-level compatibility checks, including regenerating and redeploying jOOQ-based services and testing against boundary values.

2. Dynamic jOOQ compilation: We are moving the order creation and shipment registration Service to dynamic jOOQ compilation, so that its type mappings stay in sync with the live database schema instead of depending on manually regenerated classes.

We are seeing higher failures in order creation and shipment registration services across customers.