Inbound Clinic Voice Agent

Booking an appointment by phone is a conversation with state — the caller gives information out of order, changes their mind, and asks questions mid-flow. A scripted IVR handles none of that, and a chatty LLM with no backend handles the conversation but cannot actually book anything. The system has to hold a conversation and mutate real scheduling data, correctly, in real time.

PSTN calls arrive through LiveKit SIP into a Pipecat pipeline, one container per session. Audio flows caller → Deepgram ASR → Claude Haiku 4.5 → Cartesia TTS → caller. When the model needs scheduling data it makes HTTP tool calls against a FastAPI backend over SQLite, so availability and bookings are real rather than hallucinated. Every stage is instrumented separately, which is what made the latency problem diagnosable.

Per-stage instrumentation made the bottleneck obvious and uncomfortable: the LLM is 71% of end-to-end latency at P50 (2,925ms of 4,098ms), while ASR and TTS are rounding errors. Optimising speech was pointless; the model call is the whole problem. P95 end-to-end reaches 5.2 seconds, which is a noticeably long pause on a live phone call — the kind of silence where callers start saying “hello?”. The backend is also SQLite on a single container, which is right for a demo and wrong for a clinic.

19 of 19 evaluation cases pass — happy paths, edge cases and adversarial scenarios — up from an 88% completion baseline. The number I care about more is the latency breakdown, because it turned “make it faster” into a specific, measurable target.