THE CHALLENGE

Voice assistants feel robotic when they finish their whole answer before letting you speak. Natural conversation means low latency, streaming everywhere, and — hardest of all — graceful interruption.

• One WebSocket: Mic capture (16 kHz PCM) up, synthesized audio (48 kHz) down — every event a JSON envelope, validated with Zod.
• Round Lifecycle: Every turn carries a UUID roundId, so both client and server can drop late events from a previous round the moment you interrupt.
• Silence Detection: 500 ms of silence finalizes the transcript and hands off to the LLM — no push-to-talk button.

HOW IT WORKS

The backend orchestrates three streaming services and keeps them all cancellable at any moment.

STREAMING PIPELINE

• Speech-to-Text: Cartesia ink-whisper streams a word-level transcript diff in real time as you talk.
• LLM: The reply streams from gpt-4o-mini via OpenRouter, buffered until each sentence boundary.
• Text-to-Speech: Each completed sentence is synthesized by Cartesia sonic and streamed to the client as PCM chunks — speech starts before the answer is finished.

INTERRUPTION HANDLING

• Instant Cutoff: Speaking while a response is in flight aborts the LLM stream, clears the TTS queue, and stops in-flight synthesis.
• Self-Discarding Chunks: Late audio from the old round carries a stale roundId and is dropped on both ends — no ghost speech.
• Zero Downtime: The state flips straight back to LISTENING and the new round begins immediately.
• No Self-Hearing: Both directions run through one AVAudioEngine graph with voice processing enabled, so the assistant never hears itself and talks over its own voice.

STATE MACHINE

• Four States: IDLE → LISTENING → PROCESSING → SPEAKING, owned by a single orchestrator coordinating the whole pipeline.
• Auto-Recovery: Any failure lands in ERROR and recovers back to LISTENING automatically — the conversation never dead-ends.
• Mirrored Client: The SwiftUI app gates its audio queue on the current roundId and clears it on every new response.

TECHNICAL APPROACH