Skip to content

feat(desktop): add renderer-owned Realtime Talk transport - #80

Open
TheAngryPit wants to merge 2 commits into
TheSmokeDev:mainfrom
TheAngryPit:feat/desktop-realtime-voice-contract
Open

feat(desktop): add renderer-owned Realtime Talk transport#80
TheAngryPit wants to merge 2 commits into
TheSmokeDev:mainfrom
TheAngryPit:feat/desktop-realtime-voice-contract

Conversation

@TheAngryPit

Copy link
Copy Markdown

Summary

Adds an opt-in Desktop Realtime Talk transport that keeps Hermes Core as the single chat authority while using a provider Realtime session only for low-latency speech input and playback.

This advances #44 and depends on the narrow Hermes Desktop composer ownership contract in NousResearch/hermes-agent#100666.

Problem

The plugin's existing Desktop flow cannot safely take microphone ownership while the built-in wake listener is active. It also cannot submit finalized speech or observe the canonical assistant response through a supported renderer contract.

Using the relay's normal automatic-response mode would create a second answer authority: the provider could answer the user directly, independently of Hermes chat, and could receive a provider tool catalog that the renderer transport does not need.

Design

The change is split into two reviewable commits inside this PR:

  1. Renderer-only session minting

    • adds a route-authorized desktop-relay session mode;
    • mints provider sessions with automatic_response=false;
    • sends an empty provider tool catalog;
    • preserves the existing route authorization and leaves other Talk surfaces unchanged.
  2. Desktop controller lifecycle

    • mounts through the existing composer.actions render contribution;
    • acquires/releases the Core voice lease and respects stale-session fencing;
    • sends finalized user speech through the canonical composer submit path;
    • waits for the canonical Hermes assistant snapshot before requesting provider audio;
    • handles barge-in with the active provider response_id and clears local/provider audio buffers;
    • ignores stale transcripts and stale assistant snapshots across restart/navigation;
    • disposes subscriptions and media resources on stop, failure, and unmount;
    • treats a rejected Core submission as an explicit failure instead of waiting forever;
    • exposes a one-click retry after startup or runtime failure.

Core owns microphone/wake/composer coordination. The plugin owns provider session minting, WebRTC, audio playback, interruption, and UI state.

Security and authority boundary

  • no provider credentials are stored in the renderer;
  • the existing backend route authorization remains in force;
  • renderer-only minting disables provider automatic responses;
  • renderer-only minting exposes no provider tools;
  • the provider voices only the already-authoritative Hermes assistant text;
  • no private operational files or local paths are included.

Testing

Focused Desktop behavior:

  • tests/test_desktop_plugin.py — 6/6 passed;
  • tests/test_desktop_turn_interruptions.py — 7/7 passed;
  • includes the rejected-submit regression.

Broader local validation:

  • plugin suite with the single known macOS path-literal test deselected — 1333 passed, 16 skipped;
  • the excluded pre-existing test rejects the substring private, which is also present in the standard macOS temporary-directory root; it is unrelated to this change;
  • Ruff — passed;
  • node --check desktop/plugin.js — passed;
  • uv build — passed;
  • git diff --check — passed.

Manual validation still required

A live microphone/WebRTC canary requires a Hermes Desktop build containing the Core dependency and explicit microphone permission. It should cover:

  1. start and stop;
  2. one spoken turn returning one canonical Hermes answer;
  3. barge-in during playback;
  4. navigation during an in-flight turn;
  5. provider/session failure followed by one-click restart.

No live service, installed plugin, credentials, or user configuration was modified while preparing this PR.

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant