Interviewer — webhook (SDS easy)
Candidate prompt: prompt. Method: Method. SDS: Design Webhook (easy).
Do not let the candidate read this file. Give numbers only when they ask.
Why this problem for this domain: agent (or Stripe-like) systems push books-and-records events. Ack fast, process async, never double-apply a notice, don’t assume order. Same skeleton as ingest, smaller board. EM still owns DLQ, on-call on workers, and per-bank fairness.
Keep this mock shorter than voting. If they fly, add fairness or out-of-order; don’t invent Kafka partitions for sport.
How to run it (30–45 min)
| Step | Time | Do not leave until |
|---|---|---|
| 1. Requirements | 6–8 min | Ack vs process are different; at-least-once; worse of lost vs double-apply vs leaked |
| 2. HTTP contract | 3 min | Return success after durable accept, not after full business logic |
| 3. High-level design | 10–12 min | Handler → durable buffer → worker → store; idempotent apply |
| 4. Deep dives | 8–10 min | Two of: duplicate event_id, crash after 200, HMAC, out-of-order, noisy neighbor bank |
| 5. EM close | 3 min | Adapter vs platform, lag/poison KPI, this domain mapping |
If time dies, skip CDN. Keep 200/202 after enqueue and event_id dedupe.
Step 1 — Requirements
Open
“Stripe, GitHub, or an agent bank POSTs an event at you. What has to be true before you return HTTP success?”
If they process the whole ledger on the request thread: “The sender times out at 2s. What happens to the notice?”
They should ask
- Who sends? Auth?
- At-least-once vs exactly-once?
- Volume? Retention?
- Ordered?
- What’s worse: drop, duplicate apply, slow ack?
Answers when asked
| Question | Give them |
|---|---|
| Senders | External providers. For this domain analog: agent servicing systems. Multiple tenants. |
| Auth | Shared secret / HMAC (or mTLS). Do not take unsigned public POSTs. |
| Guarantee | If we return success, we must not lose it. Processing is at-least-once. Effect must be idempotent. |
| Volume | SDS-style: ~1M events/day, 5× peak. this domain is smaller; still design for burst (notice storm), not average. |
| Ack latency | Fast (hundreds of ms). Full apply can be slower if the sender already got success. |
| Retention | Original payload + result. SDS says ~30 days; this domain audit may be years — they can say “longer for money events.” |
| Order | Not guaranteed. Paid-before-created happens. |
| Failure preference | Lost after ack is a trust incident. Duplicate apply of a paydown is also a trust incident. Slow 503 that makes them retry is better than a silent 200 and a crash. |
Gate
They have split accept from apply, and named duplicates.
Weak: Write DB on the request, return 200, done.
Strong: Durable queue (or outbox) then 200/202; workers apply with event_id.
Step 2 — HTTP contract
Want: POST /webhooks/{source} (or /webhook). Authenticate. Enqueue/persist accept. Return 2xx meaning accepted. 401/403 on bad signature. 429 if they are flooding. 5xx if we did not accept — sender retries.
202 Accepted is a good EM comment; 200 is what many providers want. Either is fine if they say accepted ≠ applied.
Gate
They do not return 200 after a 30s business transaction on the inbound POST.
Step 3 — High-level design
One box at a time.
- Load balancer + webhook handlers (stateless)
- Auth (HMAC) before enqueue
- Durable queue / log
- Workers
- Event store (raw payload + status) + golden/business DB
- Idempotency on
event_id - DLQ + metrics (queue depth, poison, lag by sender)
Push
- “Walk a retry of the same event.”
- “Worker dies after DB write, before ack to the queue.”
- “One bank sends 10× the others.”
Traps
| Trap | Follow-up |
|---|---|
| 200 then enqueue | “Crash between 200 and enqueue. Lost?” |
| Unique on payload hash | “Two legitimate notices, same body shape?” Need provider event_id. |
| One queue, no tenant key | Fairness. |
| Process as source of truth without fetching | Out-of-order: paid before created — fetch latest from source if you can, or last-write with timestamps. |
Gate
Accept path is short. Apply is idempotent.
Step 4 — Deep dives (pick 2)
A. Duplicates
Store processed event_id. Second delivery is a no-op success. Queue at-least-once + unique constraint.
B. Crash after 200
200 only after enqueue/persist. If enqueue fails, 5xx, they retry.
C. HMAC / replay
Verify signature on raw body. Optional timestamp skew to blunt replay. IP allowlist is extra, not enough alone.
D. Out-of-order
Do not assume sequence. Prefer source-of-truth read (provider/agent API) or skip older timestamps. Same as “don’t apply a stale position over a newer one.”
E. Noisy neighbor
Per-sender queue or partition + rate limit. One agent must not stall all ingest (this domain go-live).
Step 5 — EM close
- v1: HMAC, enqueue-then-2xx, event_id, one worker pool, Datadog on lag/poison. Not a custom Kafka mesh per bank.
- Topology: ingest/platform owns the hose; verticals consume facts. Mapping stays at the adapter boundary.
- KPI: accept success, apply lag p95 by agent, duplicate-apply = 0, DLQ age
- People: page workers, not the SPA. Newest engineer does not own unsigned webhooks alone.
- Exec: “We ack that we have the notice, then we apply it honestly. We will not 200 and pray.”
- this domain: this is the agent push path; Phia ingest off the request path.
Rubric
| Weak | Hireable | Strong | |
|---|---|---|---|
| Requirements | Sync process on POST | Queue | Accept vs apply; lost-after-ack is P0 |
| Design | One box | Handler + queue + DB | event_id + HMAC + lag by sender |
| EM | “We’ll add Kafka” | Mentions retries | Fairness, on-call, this domain hose |