Interviewer — agent data ingest (DDP)
Candidate prompt: prompt. Method: Method.
Do not let the candidate read this file. Give numbers and product rules only when they ask, or when they are stuck after you have nudged them to ask.
This is an Engineering Manager loop. This problem is golden source, adapters, and honesty when the hose is late — not Twitter ingest.
How to run it (45–60 min)
| Step | Time | Do not leave until |
|---|---|---|
| 1. Requirements | 8–10 min | Agent is source of truth; no invented rows; permissioned read; late-feed behavior |
| 2. APIs | 3–5 min | Ingest/event vs query; authz on query |
| 3. High-level design | 12–15 min | Adapter at boundary, queue, normalize, golden store, entitlements, query |
| 4. Deep dives | 12–15 min | Two of: late/partial feed, poison, backfill/replay, hot partition, rate limits |
| 5. EM close | 3–5 min | Adapter capacity vs next bank go-live; platform vs app; lag SLO |
If time dies, skip CDN. Keep do not invent rows and entitlements on every read.
This mock can be shorter (30–40 min) if they already did voting/VRM. Then: requirements + HLD + one failure dive + EM close.
Step 1 — Requirements
Open
“How would you approach getting agent data onto the platform so lenders can trust it?”
If they start with Kafka: “What is true before we have a cluster?”
They should ask (nudge if silent)
- Push from agent vs pull?
- What entities? (facility, position, cash flow, notice)
- Real time meaning?
- Who may read?
- What if the agent is down or late?
- New agent onboarding?
- Audit / retention?
- What’s worse: stale, wrong, down, leaked?
Answers to give when asked
| Question | Give them |
|---|---|
| Source | Agent servicing systems via API / files they provide. Not email scrape as the architecture. |
| Truth | Agent books and records are golden. We normalize labels; we do not invent positions. |
| Entities | Facility, position (often by legal entity/fund), cash flow, notice. Transaction history. |
| Delivery | Mix: some agents push events; some we poll; some send files. v1 per agent can differ. Adapter stays at the boundary. |
| Real time | Near-real-time when the agent is sending. UI must be honest if we are behind. “Real time” is not a guessed number. |
| Readers | UI and bank APIs. Strict permissioning: a lender sees only entitled facilities/positions. |
| Scale | ~11k facilities, ~$4.8T. Ingest is per-agent hose + bursts (notices, end of day), not millions of tweets. Query path is denser (grids). |
| Onboarding | New agent is a go-live. Historical backfill + then incremental. Don’t treat all agents as one schema. |
| Audit | Append-only history of what we applied. Banks will ask “why did this position change?” |
| Failure preference | Wrong/leaked worse than showing last good + delayed. Serving a fabricated position is a trust incident. |
| Cloud | Azure. Queue/event tech of their choice if they can defend it. |
| Rate limits | Agents will rate-limit you. Do not hammer them on reconnect. |
Gate
Do not start APIs until they have said, in some words:
- Agent schema ≠ platform schema (mapping)
- Durable ingest
- Permissioned query
- Behavior when behind (honest UI)
Weak: Scrape email; one giant table; cache as the product. Strong: Per-agent adapter; idempotent apply; last-good + delayed; entitlements on read.
Step 2 — APIs
Two surfaces:
Ingest (agent-facing / connector): webhook or pull cursor; stable event ids; backfill cursor.
Query (UI + bank API): facilities, positions, notices, history; filters; as_of / sync state; never a row the caller isn’t entitled to.
If they only design ingest: “What does the lender API return while the queue is 40 minutes behind?”
Gate
Ingest + query, and authz on query.
Step 3 — High-level design
One box at a time.
Boxes you want (reveal order)
- Agent connectors — per-agent adapters; their schema stays at the boundary
- Ingest queue — agent slowness/burst must not take down query API; at-least-once → idempotent consumers (stable event id)
- Normalize / mapping — golden model
- Golden store (strong consistency for positions) + append-only audit
- Entitlements service — checked on every read, API and UI
- Query API — replicas/cache keyed with as_of / version, not as naked truth
- Observability — lag by agent, poison rate, mismatch samples
Push questions
- “Walk an interest notice from agent to a lender screen.”
- “Two events, same id, delivered twice. What is the position?”
- “Agent B uses different facility labels than Agent A.”
- “Cache hit on a position used for a later vote — is that OK?”
Traps
| Trap | Follow-up |
|---|---|
| One canonical schema in the adapter for all banks | “Who loses when the fifth agent doesn’t fit?” |
| Query service writes guesses on timeout | “Write down what the user sees.” |
Cache without as_of | “That position is now a voting input. Still OK?” |
| Equal staffing of every adapter | Go-lives are sequenced. Capacity is a design constraint. |
Gate
Hose is decoupled from query. Mapping is explicit. Entitlements exist. They can describe a late agent without inventing data.
Step 4 — Deep dives (pick 2–3)
A. Late / partial feed
Want: last good snapshot + agent delayed (or equivalent) in API/UI. Never a blended guess. Partial file: don’t apply half a netting as if complete unless they define a valid partial.
B. Poison message
Want: DLQ, hose keeps moving, alert with agent+event id, replay after fix. One bad notice must not stall the bank.
C. Reconnect / backfill
Want: cursor/offset; idempotent apply; don’t double-apply notices; backoff so you don’t DDoS the agent; backfill vs realtime as two modes.
D. Hot partition
Want: discuss facility_id vs agent_id. Mega-facility + vote day = hotspot. Don’t pretend hash(agent) is even.
E. Entitlement on query
Want: UI and API agree; negative tests; support tooling still scoped.
Stakeholder curveball: “Three banks’ full history this weekend, and voting Monday.” Load shaping: don’t. Sequence go-lives vs deadlines.
Step 5 — EM close
Listen for:
- v1: one more agent (or one more entity type) with honest lag, not all agents rewritten
- Topology: connector/platform squad vs product verticals; mapping ownership decided in design review, not Slack
- Vendors: each bank is a partner/rate-limited system of record — Temelio instinct
- KPI: ingest lag p95 by agent; poison rate; entitlement false-allow = 0; escaped wrong-row
- People: don’t staff all adapters equally; next go-live wins; on-call on workers
- Exec: delayed-and-honest vs on-time-and-wrong
Phia parallel (candidate): ingest off request path, coalesce, respect rate limits. Difference: Phia could serve stale listings; here wrong is worse.
Rubric
| Weak | Hireable | Strong | |
|---|---|---|---|
| Requirements | Email scrape / “just Kafka” | Agent as source, queue | Honest lag; no invented rows; entitlements |
| Design | One DB from the webhook | Adapter + queue + store | Idempotent apply, audit, as_of cache |
| EM | Hire more for every bank | Mentions onboarding | Capacity vs go-live date; lag SLO; mapping ownership |