Idempotency

By Thomas Edwin Santosa · March 1, 2026

In distributed systems, any message can be lost, delayed, or delivered more than once. If a client doesn’t get a response, it retries — and the server must handle that retry without charging a customer twice, sending two emails, or creating duplicate records. The property that makes this safe is idempotency.

Mathematical Definition

Benjamin Peirce formalized the term in 1881: an element a of an algebra is idempotent if $a \cdot a = a$ ^[1].

Applied to functions, $f$ is idempotent when $f(f(x)) = f(x)$ for all $x$. Familiar examples:

• The identity function: $\operatorname{id}(\operatorname{id}(x)) = \operatorname{id}(x)$
• The absolute value: $\lvert \lvert x \rvert \rvert = \lvert x \rvert$
• Any constant function: $f(x) = c$ implies $f(f(x)) = f(c) = c$

In Computer Science

RFC 9110 defines an HTTP method as idempotent when “the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request” ^[2]. PUT and DELETE are idempotent; POST is not.

Pat Helland frames it more broadly: messages can be retried, reordered, and redelivered, so idempotent processing is essential ^[3]. This shifts idempotency from a nice-to-have to a hard requirement for any system that communicates across process boundaries.

Idempotency Does Not Compose Automatically

A tempting assumption: “if my sub-operations are idempotent, the whole pipeline is idempotent.” This is false — idempotency is not closed under composition.

Counterexample 1 — a composite can be idempotent even when a sub-operation is not:

Let $g(x) = 0$ (idempotent) and $h(x) = x + 1$ (not idempotent). Their composition $f = g \circ h$ gives $f(x) = 0$, which is idempotent.

Counterexample 2 — two idempotent operations composed may not be idempotent:

On $\mathbb{R}^2$, let $p(x, y) = (x, 0)$ and $q(x, y) = \left(\frac{x + y}{2}, \frac{x + y}{2}\right)$. Both are idempotent projections. Define $r = p \circ q$.

Apply it again:

Since $r^2(x, y) \neq r(x, y)$ in general, $r$ is not idempotent.

Helland’s CIDR paper makes the operational requirement explicit: the recipient entity must be “prepared in its durable state to be assailed with redundant messages that must be ignored” ^[4].

A payment flow in practice

Consider a three-step pipeline where each step uses a different dedup strategy:

1. Insert payment record — database PK (permanent)
2. Charge payment gateway — the gateway’s own idempotency key
3. Send confirmation email — Redis lock + idempotency key (TTL)

If step 2 fails after step 1 succeeds, the retry is safe: step 1 is a no-op (PK exists), step 2 deduplicates (gateway key), and step 3 won’t double-send (Redis check). But if step 3’s TTL expires before a very late retry, the email sends twice — illustrating how TTL-based dedup weakens the composite guarantee.

Takeaway: making each sub-operation idempotent is the most reliable strategy, even though it is mathematically neither necessary nor sufficient. Alternatively, a durable execution platform like Temporal can manage composition at the workflow level (see below).

Practical Strategies

1. Database primary key (no TTL)

Insert a row with the idempotency key as the primary key (or under a unique constraint). If the insert conflicts, the request is a duplicate. Permanent, simple, reliable.

Key derivation with UUIDv5: the client sends an arbitrary idempotency key (any string). The server derives the PK deterministically:

pk = uuid5(NAMESPACE_SERVICE, client_idempotency_key)

This normalizes arbitrary client strings into a valid UUID, and namespace isolation means the same client key won’t collide across different services ^[5].

If this key is on the canonical business row (for example, the transaction/payment record itself), dedup state follows normal business retention and usually doesn’t need separate idempotency cleanup. Growth management mainly applies when you keep a separate idempotency registry table for cached responses/metadata.

2. Kafka idempotent producer

Kafka assigns each producer a producer ID and tracks sequence numbers per topic-partition. The broker rejects any message whose sequence number isn’t exactly one greater than the last committed — duplicates return an error that the producer silently ignores ^[6].

This is mostly transparent to application code (enable.idempotence=true, and enabled by default in recent Kafka releases), but the guarantee is scoped to producer retry behavior within a producer session. Cross-partition atomicity still requires Kafka transactions.

3. Redis lock + idempotency key (with TTL)

A three-phase pattern ^[8]:

1. Acquire lock: SET lock:{key} owner NX EX 30 — short TTL prevents race conditions (two concurrent requests with the same key) and auto-releases on crash.
2. Execute the operation.
3. Store result, then release lock safely: first SET idempotency:{key} result EX 86400 (longer TTL for dedup), then release the lock only if you still own it (token compare + delete via Lua).

The short lock TTL handles crash recovery. The longer idempotency key TTL handles retries over hours or days. Risk: after the idempotency key TTL expires, a very late retry is re-processed.

4. Temporal workflow-level idempotency

The Workflow ID itself acts as an idempotency key — starting a workflow with a duplicate ID returns an error or joins the existing execution ^[7]. Activities run with at-least-once delivery and automatic retries; you make each activity idempotent, and Temporal guarantees the workflow completes exactly once.

This is the “let the platform handle composition” approach: instead of manually wiring lock → operation → key, Temporal’s durable execution model manages retries and dedup at the workflow level.

Comparison

Try the Simulations

Request-Level Simulator (DB vs Redis)

Switch between Database PK and Redis modes to see how TTL expiry changes behavior. In this demo, Redis mode uses ~300ms processing and an 8s idempotency TTL to model a short operation with a larger deduplication window.

What to observe

1. Database PK mode: re-send the same key — always duplicate. Permanent.
2. Redis mode: send a request and notice the row is processing... briefly, then turns into processed (Ns) with a TTL countdown.
3. Redis mode: after the countdown expires, the same key is processed again — the dedup window has closed.
4. Redis mode: send a request, then immediately re-send the same key while the lock is held — you’ll see locked (race condition prevented).

Now apply the same ideas to a composed 3-step payment flow, including Temporal workflow-level dedup.

Payment Flow Simulator (Manual vs Temporal)

What to observe

1. In Manual Composition mode, a fail-once in gateway or email produces a partial attempt, and retries replay each step with idempotent no-ops where applicable.
2. In Manual Composition mode, email dedup is TTL-bounded; after expiry, a late retry can send another email even when DB and gateway are duplicates.
3. In Temporal mode, starting the same workflow key again joins the existing/completed run, so charge/email side effects stay single-execution at workflow scope.

References

1. B. Peirce, “Linear Associative Algebra,” Amer. J. Math., vol. 4, no. 1, pp. 97–229, 1881. JSTOR ↩
2. R. Fielding et al., “HTTP Semantics,” RFC 9110, §9.2.2, IETF, June 2022. IETF ↩
3. P. Helland, “Idempotence Is Not a Medical Condition,” Commun. ACM, vol. 55, no. 5, pp. 56–65, May 2012. CACM | ACM DOI ↩
4. P. Helland, “Life beyond Distributed Transactions: an Apostate’s Opinion,” CIDR, 2007. ACM Queue | Original PDF ↩
5. “UUID Version 5,” RFC 9562, §5.5, IETF, May 2024. IETF ↩
6. “KIP-98 — Exactly Once Delivery and Transactional Messaging,” Apache Kafka. Apache Wiki ↩
7. “Temporal Workflows” and “Idempotency and Durable Execution,” Temporal.io. Docs | Blog ↩
8. “What is idempotency in Redis?,” Redis Blog. Redis ↩