← Explore learning paths

For Python developers

Building Real-Time Voice Agents from Scratch

Explore how voice agents listen, respond, and handle interruptions, with attention to latency and audio flow.

8 lessons ready to readFree to readLearn at your own pace
Read the first lesson → Sign in to track your progress →
01 / READ

Start with a lesson that meets you where you are.

02 / PRACTICE

Try questions to check what you understand.

03 / RETURN

Sign in to keep your practice progress together.

Lessons

8 of 42 topics have lessons available today. Choose any available lesson.

  1. 01

    Part I: Foundations

    Read the lesson, then try the practice.

  2. 02

    Shape of a Voice Agent

    Read the lesson, then try the practice.

  3. 03

    Audio Fundamentals

    Read the lesson, then try the practice.

  4. 04

    VAD: Detecting Speech

    Read the lesson, then try the practice.

  5. 05

    Part II: The Pipeline

    Read the lesson, then try the practice.

  6. 06

    Part III: The Hard Parts

    Read the lesson, then try the practice.

  7. 07

    Part IV: Engineering It Well

    Read the lesson, then try the practice.

  8. 08

    Part V: Make It Yours

    Read the lesson, then try the practice.

Topics without lessons yet

These are part of the outline. Lessons are not available for them yet.

  • mic → ASR → LLM → TTS Loop
  • Trade Matrix
  • SR_IN vs SR_OUT
  • float32 ↔ int16 Conversions
  • Threshold Tuning
  • Pre-roll Buffer
  • ASR with faster-whisper
  • Model Size Trade-offs
  • ASR as a Blocking Call
  • LLM Streaming & State
  • Speakable System Prompt
  • The Commit Pattern
  • TTS & Latency Trick
  • pop_sentences Deep Dive
  • Kokoro vs Piper Backends
  • Barge-in: Interruption
  • Yield-Point Latency
  • Cancel Wire Protocol
  • The Feedback Loop
  • Browser AEC
  • Playback State Machine
  • Three Distinct Moments
  • Frontend Audio Scheduling
  • AudioWorklet for Mic Capture
  • Gapless playHead Scheduling
  • Concurrency & Orchestration
  • run_in_executor Pattern
  • asyncio vs Threads — Same Shape
  • Capstone Extensions
  • Measurable Latency Fork
  • Extension Projects
  • The Production Bridge
  • Trade-offs You Now Own
  • Why Hosted APIs Choose as They Do