Microphone bytes to spoken words.

01 Heritage: unmute, rebuilt.

The bones of the pipeline come from Kyutai's unmute — their open-source reference for a fully self-hosted speech-to-speech assistant. The state machine (waiting_for_user / user_speaking / bot_speaking), the msgpack wire protocol for the Moshi STT/TTS servers, and the VAD-pause-detection model are all from there. What BawtHub layers on top is:

• A multi-provider TTS registry — Moshi is no longer the only TTS option; Azure, xAI Grok, and Kokoro can serve different voices in the same session.
• An llm-bawt back end instead of unmute's bundled chatbot — meaning real conversation history, real memory, real tools, real personalities.
• A Quest-based supervisor for the STT/TTS/LLM workers — co-managed lifecycles, retries with backoff, structured cancellation.
• Production niceties — health endpoints, admin restart, voice browser, provider auto-discovery from env, container restart through a mounted Docker socket.

02 The pipeline, top to bottom.

03 The handler is the state machine.

BawtHubHandler in bawthub/handler.py is an AsyncStreamHandler from fastrtc. It runs one instance per connected client and owns three things:

• A QuestManager with three named workers: stt, tts, llm. Each Quest has init / run / close phases and survives partial failures.
• A Chatbot with a small chat_history array. Conversation state is derived: waiting_for_user / user_speaking / bot_speaking based on whose turn it is in the history.
• An asyncio output_queue that fastrtc drains to emit either audio frames, OpenAI Realtime events, or a CloseStream.

The receive loop is the heart of it. Every frame from the browser (~80 ms of audio) gets:

1. Forwarded to STT via stt.send_audio(array)
2. Checked against the long-silence timer for nudge behavior
3. Tested for pause: if stt.pause_prediction.value > PAUSE_THRESHOLD (default 0.6), the handler emits InputAudioBufferSpeechStopped, flushes 500 ms of silence into STT, and starts the LLM response
4. Tested for interrupt: if the bot is speaking and STT-VAD shows the user has resumed (and we're past the 3-second grace window), interrupt_bot() cancels the running LLM + TTS quests

04 STT: Kyutai Moshi over msgpack.

The STT runs in its own GPU container — Kyutai's moshi-server with the stt.toml config:

The wire is msgpack over a websocket at /api/asr-streaming. The Python client (bawthub/stt/speech_to_text.py) sends raw float32 audio frames and receives a typed stream of:

• Word — a partial transcription token with start_time
• EndWord — explicit end-of-word marker
• Step — per-step pause-prediction probabilities (the VAD signal)
• Marker — for time-aligning events
• Ready / Error

The handler tracks stt.pause_prediction as an exponential moving average so the threshold check is stable across noisy frames. The "real" pause threshold is configurable via BAWTHUB_PAUSE_THRESHOLD; defaults to 0.6.

05 TTS: a provider registry, not a single engine.

BawtHub's TTS layer evolved from "Moshi only" to a pluggable registry. TTSProviderRegistry in bawthub/tts/registry.py manages a set of BaseTTSProvider instances and dispatches each request to the right one based on voice id.

Provider selection is voice-driven. The registry inspects each provider's available_voices catalog and routes a request for voice_id to the provider that owns it. BAWTHUB_TTS_DEFAULT_PROVIDER overrides the tiebreaker for voice ids that aren't in any catalog. Auto-discovery is conservative: a provider that fails to construct (e.g. missing SDK, bad credentials) is dropped silently so the app still boots in a stripped-down environment.

Health probing is lazy and cached

Construction does not block on a network probe. The first request that asks "is this provider healthy?" pays the probe cost; subsequent requests within HEALTH_TTL_SEC (5 s default) read the cached result. Concurrent probes share a single in-flight check via an asyncio lock — so a thundering herd at startup doesn't slam Azure with twenty list_voices calls.

06 Word-by-word TTS with a prebuffer.

llm-bawt streams the LLM response word-by-word (one delta per word, no upstream sentence buffering). The handler's _generate_response_task pipes each delta into the TTS provider with one twist:

The TTS provider streams back two message types: TTSAudioChunk (PCM at 24 kHz, mono) and TTSWordTiming (which word just got spoken, with start/stop seconds). The handler routes audio chunks to the output queue as (sample_rate, np.ndarray) tuples, and word timings as ResponseTextDelta events that the browser uses to update the floating transcript. Each word is also appended to chat_history only after the TTS confirms it — so the displayed transcript advances in lockstep with the audio.

07 Mid-word interrupts.

The user can interrupt the bot at any time. Two signals trigger an interrupt:

1. STT word arrival while bot is speaking. A confident transcription token from the user during bot output → interrupt_bot().
2. VAD probability dropping past a threshold while bot is speaking, after the 3-second grace window.

interrupt_bot() cancels the LLM stream task, swaps the handler's output_queue to a fresh queue (so any in-flight TTS workers can't accidentally push to it), and sends BawtHubInterruptedByVAD to the browser. The browser stops playback by signalling the AudioWorklet to drop its buffered samples. Audio cuts mid-word — which is what feels conversational.

08 Wire protocol: OpenAI Realtime, deliberately.

The browser ↔ backend WebSocket speaks OpenAI Realtime API events, mapped to BawtHub's own ora module (openai_realtime_api_events.py). The choice is intentional — it keeps the door open to swap the backend for OpenAI's actual Realtime API or any compatible implementation.

09 About wake words.

BawtHub doesn't ship a wake-word detector. The voice page assumes the user has pressed the call button — the WebSocket is opened intentionally, the screen is wake-locked, and the mic is live until the call ends. Pause detection is purely VAD-driven via Moshi STT's extra-head pause_prediction scores.

That said, the downstream consumers of BawtHub's voice (Home Assistant via the bawthub_stt / bawthub_tts custom components in integrations/home_assistant/) are typically paired with ESPHome Voice PE devices, which run their own on-device microWakeWord models. So while wake-word handling isn't inside BawtHub itself, the pipeline is designed to be a downstream of a wake-word gate when used as a Home Assistant voice provider.

10 Health and admin.

The voice backend exposes a handful of REST endpoints distinct from the WebSocket:

The TTS admin state lives at bawthub/tts_admin_state.json — a tiny JSON file on disk so changes survive container restarts. The frontend's /tools/tts page is the editor.

11 Key files.