Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,36 @@ named rather than smoothed.
## [Unreleased]

### Added
- `pause_voice_input` — the model can pause listening without ending the
call (#100). "Stop listening" or "mute the mic" is a tool call: the session
stays connected, playback keeps playing, background work keeps running and
its results are still announced; only the operator's speech stops reaching
the provider. The flag lives on the capture surface — `DuplexAudio` and
`DiscordAudio` grew `pause_input` / `resume_input` / `input_paused`, the
same one-interface pattern as `playback_pending` in #87 — so both rooms
honour it identically: blocks captured while paused are dropped (never
queued stale), the already-queued ones are discarded, and the Discord bridge
keeps the host's buffers drained and its inactivity timer armed so the bot
is not evicted from the channel. A paused microphone cannot hear the word
"resume", so the way back is the operator's own control — Enter in the
standalone `hermes talk` terminal (toggle; `p`/`r` explicit — a polling
watcher, never a blocking stdin read), `/talk pause` / `/talk resume` in
Discord (`/talk status` says when it is paused) — and the tool is offered
ONLY where that control is guaranteed: the pause decision is made once,
before the tool list is built, from the same predicate that starts the
keyboard watcher, and the registered control is what the receipt names.
No control, no pause: a piped or non-tty stdin gets no key and no tool;
`/talk` at the Hermes prompt shares its tty with prompt_toolkit, so that
lane never watches stdin and offers no pause; and a pause call that
arrives anyway is refused (`no_resume_path`) rather than armed, because a
pause nobody can undo would be a hang-up. On Windows an extended key
(arrows, Insert, F-keys) is consumed whole — before, Down-Arrow's scan
code read as `p` and paused the microphone. Both directions get a spoken
receipt: the model's tool result for its own flips, a contained
announcement for the operator's. The tool classifies read-only (it can
only narrow what a session does, and a pause is never a path to authority)
and refuses — never arms — when no session is attached. Ported idea from
bielcarpi/hermes-live-voice (MIT), idea only.
- Run admission control on `delegate_task` (#101). The model may declare
`execution_mode` (`exclusive`, the default, or `parallel_read_only`) and up
to eight normalized `resource_keys` naming what a task touches — a repo
Expand Down
27 changes: 26 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -390,14 +390,39 @@ agent-loop-only tools (`memory`, `session_search`, `honcho_search`,
says which of the two it answered from — a recollection can be stale in a way
a verbatim line cannot, and nothing is on screen to check it against.

**Pause the microphone without hanging up.** Say "stop listening" (or "mute
the mic") and the model calls `pause_voice_input`: the call stays connected,
playback keeps playing, background work keeps running and its results are
still announced — only your speech stops reaching the provider. A paused
microphone cannot hear the word "resume", so the way back is your own control,
and **the pause is offered only where that control is guaranteed to exist**:

- **`hermes talk` in a real terminal** — **Enter** toggles (`p` and `r` are
explicit; on Windows they are single keys, elsewhere type the word and
Enter). The connected line says `Enter to pause or resume the microphone`
when the key is live. With a piped or non-tty stdin (Git Bash's mintty
reports no tty to Python; launcher wrappers) there is no key, so no pause is
offered and a pause call is refused with a receipt that says why.
- **`/talk` typed at the Hermes prompt** — the prompt owns that terminal for
the whole call, so the session never watches it for a key, and offers no
pause either. Use `hermes talk` on its own when you want the control.
- **Discord** — `/talk pause` and `/talk resume`, typed. The model-side tool
is offered on the legacy provider-owned lane; on the `provider-host-tools`
lane the host supplies the tool list and the typed commands are the path.

Both directions get a spoken receipt, the receipt names the control for the
room you are in, and Ctrl+C still hangs up.

**In Discord**, `/talk join` runs the call in the voice channel Hermes is
already in — same conversation, same tools, same steering, in a room other
people can hear. Talk now reports speaker transitions to the model using the
member's immutable Discord user ID; display names are quoted as untrusted data,
and an unknown SSRC stays unresolved and unauthorized. Configure immutable IDs
with `TALK_DISCORD_OPERATOR_USER_IDS=<id>[,<id>...]`. Only those speakers may
run `delegate_task`, `steer_agent`, `redirect_agent`, or `stop_work`; everyone
may still converse and use read-only tools. Unset, blank, or any malformed list
may still converse and use read-only tools — including `pause_voice_input`,
which can only narrow what the session does; `/talk resume` (text) brings
listening back. Unset, blank, or any malformed list
authorizes nobody. Talk binds permission to the exact Discord PCM, VAD input
item, and opaque Realtime response metadata — never a display name, SSRC,
model argument, or whichever person spoke most recently. Mixed, missing, or
Expand Down
22 changes: 17 additions & 5 deletions __init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,9 @@ def _register_talk_command(ctx) -> None:
"description": (
"Start provider-owned voice with canonical Hermes tools (join), or the "
"canonical core voice lane (core join); "
"gateway also supports leave and status"
"gateway also supports pause, resume, leave and status"
),
"args_hint": "[join|core join|leave|status]",
"args_hint": "[join|core join|pause|resume|leave|status]",
}
if contextual:
kwargs["invocation_context"] = True
Expand Down Expand Up @@ -195,21 +195,33 @@ def _talk_command(raw_args: str = "", invocation=None) -> str:
try:
asyncio.get_running_loop()
except RuntimeError:
if sub in {"join", "core join", "leave", "status"}:
if sub in {"join", "core join", "pause", "resume", "leave", "status"}:
return (
"Those are for the gateway's Discord voice channel. Here in a "
"terminal, plain `/talk` starts the call."
"terminal, plain `/talk` starts the call; the standalone "
"`hermes talk` command adds Enter to pause and resume the "
"microphone."
)
# This prompt owns the terminal for the whole call (prompt_toolkit,
# raw mode, its own stdin reader), so the session must not watch
# stdin for the pause key — and without that key it offers no pause
# (hermes-talk#100). `hermes talk` on its own is the lane that does.
return (
"Voice session ended."
if talk_cli.cli_entry() == 0
if talk_cli.cli_entry(keyboard_control=False) == 0
else ("Voice session ended with errors — see stderr.")
)

if sub in {"leave", "stop", "hang up"}:
return talk_discord.stop_session()
if sub == "status":
return talk_discord.session_status()
# The room's microphone control (hermes-talk#100): text, because a paused
# session hears nobody and the way back cannot be spoken.
if sub in {"pause", "mute"}:
return talk_discord.pause_session()
if sub in {"resume", "unmute"}:
return talk_discord.resume_session()
if sub == "core join":
if not talk_core_realtime.core_provider_available():
return "Canonical core voice is unsupported by this Hermes host."
Expand Down
2 changes: 2 additions & 0 deletions dashboard/plugin_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,8 @@ def _mint(auth_token: str, voice: str, *, text_output: bool = False):
text deltas back through the cascade relay to be spoken server-side.
"""

# The browser owns this lane's microphone, so the pause tool is not
# offered here (default_talk_tools' pausable stays False).
tools = talk_tools.default_talk_tools()
return talk_wire.mint_ephemeral_session(
auth_token=auth_token,
Expand Down
4 changes: 3 additions & 1 deletion docs/OPERATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -340,7 +340,9 @@ SSRC alone, model arguments, and "last speaker" state are never authority.
Configured IDs may run the four state-changing tools: `delegate_task`,
`steer_agent`, `redirect_agent`, and `stop_work`. Other speakers retain normal
conversation and the read-only tools (`search_memory`, `search_vault`,
`check_work`, `list_agents`, `talk_status`, and `talk_capabilities`). Missing
`check_work`, `list_agents`, `talk_status`, `talk_capabilities`, and
`pause_voice_input` — a pause changes nothing outside the session and can only
narrow what it does; `/talk resume`, typed, brings listening back). Missing
response correlation,
an unresolved speaker, two speakers in one VAD turn, or a speaker outside the
allowlist returns a non-sensitive spoken denial without running the handler.
Expand Down
1 change: 1 addition & 0 deletions docs/VOICE-COMMANDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ delivery; "landed" only ever follows a real delivery artifact.
| "once" / "this session" / "no" (answering an approval question) | `resolve_approval` | "approved — just this once", "approved for the rest of the run", or "denied — the agent was told no" | voice can grant `once`, `session`, or `deny` — **never `always`** (narrowed in code, not in the prompt); an unanswered question denies itself on a timer, and interrupting the question denies it on the spot |
| "what are you running on?" / "status report" | `talk_status` | version, model, voice, auth lane, agent lane, audio, identity sections | the verification command — field-by-field meaning in [OPERATING.md](OPERATING.md#2-talk_status--the-one-command) |
| "what can you do right now?" / "which tools do you have?" | `talk_capabilities` | installed skills, resolved toolsets with their enabled/configured flags, gateway feature flags, live run counts | live evidence, not the prompt — read in-process off the attached agent, or over the api server when detached; a toolset listed `enabled: false` is reported as installed but NOT usable |
| "stop listening" / "mute the mic" / "hold on, I'm talking to someone" | `pause_voice_input` | "microphone paused — press Enter when you want me back" (standalone `hermes talk` in a real terminal) or "… say `/talk resume`" (Discord) | the call stays up: playback, background runs and their announcements continue; nothing you say reaches the provider until YOU resume it — a paused mic cannot hear "resume", so the way back is a key or a command, never speech, and the tool is offered only where that key or command exists (not for `/talk` at the Hermes prompt, not with a non-tty stdin). Resume gets its own spoken receipt |

Things you'll hear without asking (v0.6+): a background agent finishing
("Background agent sa-… finished…"), a steering note landing ("the
Expand Down
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ py-modules = [
"talk_steer",
"talk_lifecycle",
"talk_operator_auth",
"talk_pause",
"talk_progress",
"talk_realtime",
"talk_openai_realtime",
Expand Down Expand Up @@ -111,6 +112,7 @@ known-first-party = [
"talk_openai_realtime",
"talk_core_realtime",
"talk_core_provider",
"talk_pause",
"talk_progress",
"talk_providers",
"talk_realtime",
Expand Down
53 changes: 53 additions & 0 deletions talk_audio.py
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,7 @@ def __init__(self) -> None:
self._dropped_playback_bytes = 0
self._played_frames = 0
self._output_level = 0.0
self._input_paused = False
self._in_stream = None
self._out_stream = None
self._pulse_webrtc = _PulseWebRtcAudio()
Expand Down Expand Up @@ -352,6 +353,13 @@ def _input_callback(self, indata, _frames, _time, _status) -> None:
input_level = _pcm16_rms(pcm)
with self._lock:
output_level = self._output_level
paused = self._input_paused
if paused:
# Paused capture is DROPPED here, not queued and skipped later: a
# block that sits in the queue through a pause would be the first
# thing sent on resume, seconds stale. (One block can still race
# the flag; resume_input drains it.)
return
if self._echo_gate_enabled and not self._pulse_webrtc.active:
output_active = output_level > self._output_active_level
echo_threshold = max(
Expand Down Expand Up @@ -415,11 +423,56 @@ def _take_playback(self, wanted: int) -> bytes:
def read_input_chunk(self) -> bytes | None:
"""One captured block, or ``None`` when the microphone has nothing yet."""

if self.input_paused:
return None
try:
return self._input.get_nowait()
except queue.Empty:
return None

def _discard_queued_input(self) -> None:
while True:
try:
self._input.get_nowait()
except queue.Empty:
break

def pause_input(self) -> None:
"""Stop feeding captured audio to the session; playback is untouched.

From the moment this returns nothing the microphone heard reaches
the wire: blocks captured from here on are dropped in the callback,
the ones already queued (captured before the flag, not yet drained
by the sender) are discarded, and the reader answers empty while the
flag is up. Playback, ``playback_pending`` and the barge-in boundary
are unaffected — a paused session still speaks, and still knows what
it has said.
"""

with self._lock:
self._input_paused = True
self._discard_queued_input()

def resume_input(self) -> None:
"""Feed captured audio to the session again, from the next block on.

Drains first: the callback's flag check and its queue write are not
one atomic step, so one block admitted just before the pause can
land after the pause's drain. It is the stalest audio there is, and
without this it would be the first thing sent on resume.
"""

self._discard_queued_input()
with self._lock:
self._input_paused = False

@property
def input_paused(self) -> bool:
"""Whether capture is paused (hermes-talk#100)."""

with self._lock:
return self._input_paused

def queue_playback(self, pcm: bytes, item_id: str | None = None) -> None:
"""Queue model audio for the speaker. Drops on overflow, never blocks."""

Expand Down
Loading