[RFC] RealtimeVoiceProvider ABC — four competing duplex-voice PRs need an interface, not a merge queue

[TheSmokeDev]

The trigger has already fired

AGENTS.md (Footprint Ladder, ~L208):

When 3+ open PRs try to integrate the same category of thing (memory backends, providers, notifiers), don't merge them one at a time — design an ABC + orchestrator, wrap the existing built-in as the first provider, and turn the competing PRs into plugins against that interface.

There are four open PRs adding realtime duplex voice, none merged, all in salvage:

• Add OpenAI Realtime voice for Discord VC #27650 — OpenAI Realtime for Discord VC (@Solvely-Colin)
• [codex] Add realtime Discord voice mode #21504 — realtime Discord voice mode (@RationallyPrime)
• feat(voice): add realtime transcription bridge #62500 — Realtime transcription bridge (@337zzang)
• feat(gateway): add generic voice_server gateway platform #27040 — generic voice_server gateway platform (@tmylk)

Plus two feature issues asking for the same capability from different surfaces: #53836 (live multimodal voice) and #20765 (browser WebRTC voice).

That's the exact pattern the doctrine says to answer with an interface rather than a merge queue.

What's missing

Core already treats voice as a first-class extension point — but only two-thirds of it:

Shape	ABC	Registration
Text → audio	agent/tts_provider.py TTSProvider	ctx.register_tts_provider
Audio → text	agent/transcription_provider.py TranscriptionProvider	ctx.register_transcription_provider
Audio ⇄ audio (duplex session)	— none —	—

Every voice surface in the tree today is STT → text turn → agent loop → TTS. That's turn-based by construction: no barge-in, no sub-second turn latency, and no way for the speech layer to invoke tools mid-utterance. A duplex realtime session is a genuinely different shape, which is why four PRs each had to invent their own plumbing and each landed somewhere different (Discord sidecar, gateway platform, desktop composer, transcription bridge).

There's also a live-API forcing function: the beta Realtime protocol is now rejected, not deprecated (The Realtime Beta API is no longer supported), so the in-tree plugins/google_meet Realtime client is currently broken against the live endpoint — see #57448 and my receipt there. Whatever shape this settles into, it needs writing once against GA rather than four times.

Proposal

A RealtimeVoiceProvider ABC in agent/, registered via ctx.register_realtime_voice_provider(...), owning only what's genuinely provider-specific:

class RealtimeVoiceProvider(ABC):
    name: str
    def is_available(self) -> bool: ...
    def get_setup_schema(self) -> dict: ...
    async def open_session(self, *, instructions: str, tools: list[dict],
                           voice: str | None) -> RealtimeSession: ...

class RealtimeSession(ABC):
    async def send_audio(self, pcm: bytes) -> None: ...
    async def events(self) -> AsyncIterator[RealtimeEvent]: ...   # audio | transcript | tool_call | turn boundaries
    async def submit_tool_result(self, call_id: str, output: str) -> None: ...
    async def cancel_response(self) -> None: ...                   # barge-in
    async def close(self) -> None: ...

The load-bearing design point, which the sweeper already praised on #62500: Hermes stays the tool and session owner. The provider is ears, mouth, and turn-taking; tool calls surface as events and results go back through Hermes's own dispatch. Nothing about a provider gets to own the agent loop.

Surfaces (CLI, gateway platform, desktop, browser) then consume ONE interface instead of each shipping a vendor client, and the competing PRs become providers or surfaces against it rather than alternatives to each other.

Why I'm filing this rather than another PR

I built the out-of-tree version first, on purpose: hermes-talk — OpenAI Realtime duplex, GA protocol, barge-in with local playback drain + conversation.item.truncate, function-call relay into Hermes tools via ctx.dispatch_tool, and background delegation whose results are spoken when they land. Standalone repo per the June-2026 policy, zero core edits, 182 offline tests, CI on ubuntu + windows.

So this proposal isn't theoretical — the interface sketch above is the seam that fell out of a working implementation, and I have something to validate any agreed ABC against before it ships. I'm not asking for hermes-talk to be merged; vendor integrations belong out-of-tree and I agree with that rule.

Happy to implement the ABC + wire hermes-talk as the first consumer, if maintainers want it. Equally happy for someone else to own it — the four PR authors above have been at this longer than I have and should have a say in the shape. Mostly I'd like the category decision made, because right now four good implementations are stuck behind a question nobody has answered.

What would be most useful: a maintainer call on (a) whether duplex realtime gets its own provider ABC at all, and (b) if so, whether the session interface above is the right granularity.

[TheSmokeDev]

One more data point for this RFC, from the official docs rather than the code:

docs/user-guide/features/voice-mode documents the current architecture as sequential turn-taking — STT (faster-whisper / Groq / OpenAI / Mistral / xAI) → full LLM inference → TTS (Edge / ElevenLabs / OpenAI / NeuTTS), with silence-based segmentation (Discord VC waits ~0.5s speech + ~1.5s silence before inference). Barge-in exists and works well within that model.

So the gap isn't voice — voice is mature here. It's specifically the duplex shape: a single bidirectional audio session where turn-taking, interruption, and tool invocation all happen inside the speech layer rather than around it. The docs make no mention of a realtime/streaming path, which matches what the four open PRs have each been building independently.

That's the case for an ABC rather than a fifth implementation: TTSProvider and TranscriptionProvider already cover the two halves of the turn-based model, and duplex is the third shape with no seam to register against.

[TheSmokeDev]

A concrete second-provider case for this RFC just landed on the plugin side: TheSmokeDev/hermes-talk#3 — a request for a Gemini Live backend (AI Studio keys, native realtime audio plus live vision). Same run-control surface and tool dispatch, different realtime transport underneath. That's exactly the seam RealtimeVoiceProvider would own, and I'd rather build the Gemini backend against this abstraction than hand-roll a second transport in the plugin.