Interviewer — payment system (SDS)

Candidate prompt: prompt. Method: Method. SDS: Design a Payment System.

Do not let the candidate read this file. Give numbers and rules only when they ask.

Why this problem for this domain: the agent (or accounting vendor) cannot share a transaction with your DB. Exactly-once is an effect (idempotency + ledger + reconcile), not 2PC. Same muscle as Temelio and as “vote accepted, downstream notify failed.”

This is still an EM loop. After the boxes, they must staff, sequence, and measure.

---

How to run it (45–60 min)

Step	Time	Do not leave until
1. Requirements	8–10 min	External provider is outside the txn; worse of double-charge vs drop vs stale balance
2. APIs	3–5 min	Create payment with idempotency key; get status; refund
3. High-level design	12–15 min	Ledger (not a mutable balance column) + provider call + in-doubt path
4. Deep dives	12–15 min	Two of: double-click retry, timeout/in-doubt, reconciliation vs provider report, refund saga
5. EM close	3–5 min	v1, ownership of ledger vs provider adapter, KPI, this domain mapping

If time dies, skip multi-region and fraud. Keep never double-charge and timeout does not guess.

---

Step 1 — Requirements

Open

“Walk me through how you’d approach moving money when the card network is not in your database transaction.”

If they start with Kafka: “What is true on a double-click before we have a cluster?”

They should ask

• Parties (card vs wallet-to-wallet)?
• Correctness bar (double-charge / lost payment)?
• Scale vs correctness?
• Balance consistency?
• Who retries — client or worker?
• Audit / retention?

Answers when asked

Question	Give them
Parties	Card (or bank) via an external processor. Wallet-to-wallet is a variant after they have the card path.
Correctness	Exactly-once effect: never double-charge, never silently lose a payment. Balances reconcilable against what the provider actually did.
Scale	Thousands of payments/sec at peak is enough to mention. Correctness under failure dominates, not an analytics firehose.
Balances	Strong. Showing spendable funds that already left is not an acceptable window.
Timeouts	Provider may have charged while your HTTP call timed out. Guessing is forbidden.
Audit	Immutable history. Years, not days.
Out of scope	Fraud ML, interchange pricing, provider matching-engine internals.

Gate

They have named: idempotent initiate, provider as a separate system, balances that cannot lie, some story for “we don’t know yet.”

Weak: UPDATE accounts SET balance = balance - X and hope. Strong: “I cannot wrap charge + ledger in one txn. Retries must be safe. Timeout is in-doubt until we reconcile.”

---

Step 2 — APIs

Want:

• POST /payments with amount, parties, Idempotency-Key
• GET /payments/{id} — includes processing / succeeded / failed / in_doubt
• GET /accounts/{id}/balance (derived, not a naked cache)
• POST /payments/{id}/refund (its own idempotency key)

If they omit the key: “The client times out and retries. What happens twice?”

Gate

Create + get status + they have said retry of create is safe or they know it isn’t yet.

---

Step 3 — High-level design

One box at a time.

Boxes you want

1. API / gateway (auth, idempotency key intake)
2. Payment service (state machine)
3. Ledger — append-only entries; balance = sum (or snapshot + entries). Not a single overwritten column as source of truth
4. Provider adapter (their schema stays at the boundary)
5. Queue/worker for capture, webhooks, retries
6. In-doubt state + reconciliation job vs provider settlement/report
7. Observability on workers and in-doubt age, not only the HTTP 200

Push

• “Walk a successful charge.”
• “Walk a retry of the same idempotency key.”
• “Walk a timeout after you sent the charge.”
• “Where does the UI get the balance during in_doubt?”

Traps

Trap	Follow-up
Distributed transaction with Stripe/bank	“They will not join your 2PC. What now?”
Queue is the source of truth	“If the worker acks twice, did we pay twice?”
Cache as balance	“Cache says $100. Ledger says $0. What does Spend do?”
Compensate by deleting the charge row	“Audit asked what happened at 14:02.”

Gate

Provider call is not in the same ACID txn as the ledger. There is an explicit unknown state.

---

Step 4 — Deep dives (pick 2–3)

A. Idempotent create

Same key → same payment id and status. Unique constraint on key. Do not create a second charge.

B. Timeout / in-doubt

Do not mark failed. Do not mark succeeded. Poll provider or wait for webhook and a nightly/settlement reconcile. User sees processing/unknown, not a lie.

C. Webhook vs you polled

At-least-once webhooks. Dedupe on provider event id. Out-of-order events: ignore stale.

D. Refund saga

Refund is a new ledger pair, not an overwrite. Idempotent. If provider refund fails, payment stays captured; compensate with retry, not silent delete.

E. this domain translation (if they don’t go there)

“Agent books are the provider. Your platform is the ledger. What is the in-doubt UI for a notice you sent but aren’t sure applied?”

---

Step 5 — EM close

Listen for:

• v1: one rail (cards), idempotent charge, ledger, in-doubt, one reconcile job. Not every payout method.
• Topology: ledger/platform vs provider adapter (vendor-shaped, like agent connectors)
• DoD: retry tests, timeout tests, reconcile catch-up
• KPI: duplicate-charge = 0; in-doubt age; reconcile breaks; escaped money defects
• Exec: “We can look instant and maybe double-charge, or show processing until the provider confirms. I recommend the second.”
• this domain: Temelio mapping layer; voting accept vs notify; never overwrite the golden source from the SPA

---

Rubric

	Weak	Hireable	Strong
Requirements	CRUD balance	Names provider + retry	In-doubt as a first-class state
Design	Queue + hope	Idempotency key	Ledger + reconcile vs provider file
EM	More retries	Mentions audit	Staffing adapter vs ledger; money KPI