Designing idempotent APIs for unreliable networks

A client sends a POST to create a payment. The server debits the account and starts writing the response. The connection drops before the response finishes. The client sees a timeout, not a success and not a failure, and does the only thing that makes sense to a retry loop: it sends the request again. Whether that produces a duplicated payment has nothing to do with how careful anyone was writing the code. It's determined by whether the API was designed to survive the gap between the server doing the work and the client learning about it.

"Just retry the request" papers over this rather than solving it. Retries are correct behavior for a client facing a network that drops packets and times out connections. The problem is that most APIs treat every incoming request as a new fact about the world, so a retried request that already succeeded gets executed again, with no way to tell a new order from the same order arriving twice because TCP lied to somebody.

The failure isn't rare, it's structural

Enterprise integrations hit this constantly: batch jobs replaying a queue after a worker crash, an ESB retrying a call that exceeded its timeout, a partner's nightly reconciliation job resubmitting a file because their side logged a 502 even though your side committed the transaction. These aren't edge cases; they're the normal operating condition of any system with retries configured anywhere in the chain, which today is nearly all of them. We've seen a client-facing order API duplicate roughly 0.3% of orders under load, purely from retried POSTs that had already succeeded server-side. At six figures of daily order volume, that's a reconciliation team's full-time job, not a rounding error.

The bug doesn't show up in testing, either. The suite calls the endpoint once, gets a 200, and moves on. Duplication only appears under real conditions: load balancer timeouts, connection resets mid-response, retry logic upstream nobody on your team reviewed. By the time it surfaces it's already in production data, and untangling which of two "identical" orders is real, weeks later, is often not fully recoverable.

Idempotency keys move the decision to where the information actually is

The fix is to stop asking "did this happen before" using the request's content, and ask it using a key the client generates once and reuses on every retry of that same logical operation: a UUID attached as an Idempotency-Key header, resent unchanged on every attempt. The server checks whether it has already recorded a result for that key; if so, it returns the stored result without re-executing anything, and if not, it executes the operation and records the key alongside the result in the same transaction as the side effect. Recording the key as a separate step afterward reintroduces the exact race you were trying to close: the server can crash between debiting the account and recording the key, so the retry executes the debit again because the safeguard wasn't durable.

A key alone isn't enough, since a client bug could reuse a key with a different payload. Store a hash of the request body alongside the key; on a repeat, compare hashes. A matching key with a mismatched body isn't a retry, it's a conflict, and should return a 409. Scope keys to (customer_id, idempotency_key), not globally, and expire records after a bounded window, 24 hours is usually enough.

Put it in the contract, not just the database

Idempotency only works as a documented guarantee, not an implementation detail one endpoint happens to support. If POST /orders honors the header but POST /refunds doesn't, integrators will assume the guarantee is uniform, retry blind on both, and get duplication on the endpoint you forgot. Every state-mutating endpoint a retry could plausibly hit needs the same contract: accept the header, honor it atomically, and return the original response rather than a generic "already processed" error. PUT and DELETE are idempotent by HTTP semantics in theory, but a PUT that increments a counter isn't safe to repeat just because the verb says so.

The organizational failure underneath the technical one

This keeps recurring in enterprise integrations because the retry logic and the endpoint it calls are usually owned by different teams, sometimes different companies, and neither side sees the other's failure handling. The integrating team assumes the API is idempotent because that's the responsible way to build a retry loop. The API team assumes clients deduplicate on their own because nobody asked them to design for it otherwise. Both assumptions are reasonable alone and wrong together, and the gap becomes visible only when a customer calls asking why they were charged twice. Put idempotency requirements in the spec itself, next to the auth scheme, not in onboarding prose nobody reads until after the incident.