I've been experimenting with three ways to make HAL talk: a local pipeline with PocketTTS, OpenAI's GPT Live, and now a combination of the two.
Kyutai's PocketTTS lets me clone HAL's voice from a reference recording. A companion computer transcribes what I say, generates a reply, then turns that text into speech for the ESP32. I can control the voice separately from the model writing the answer, but I have to connect and coordinate each stage.
The second experiment uses GPT Live, released in the API on September 10, 2026. The ESP32 sends microphone audio and receives spoken replies directly over Wi-Fi, without a Mac companion. Live handles listening and speaking in the cloud, with a backend model for deeper reasoning. My setup uses its Meridian voice, prompted to speak with HAL's calm, measured delivery.
The catch is voice cloning. As of writing this, custom voices require separate access, so I can't simply give Live a HAL clip as I do with PocketTTS.
The third experiment brings the Mac companion back. HAL sends microphone audio through the gateway to GPT Live, which handles the conversation. The gateway discards Live's generated audio and feeds its response text to PocketTTS in short chunks. PocketTTS speaks those chunks using the HAL reference recording, then streams the audio back to the ESP32.
This gives me the reference voice with Live handling the conversation, but the extra synthesis step adds delay. Echo cancellation keeps the microphone open during playback, and recognized interruptions clear the queued speech. I'm still tuning the timing and audio flow between the board and companion.