Skip to content

feat(realtime): adapt semantic endpointing controls - #107

Open
kvnloo wants to merge 2 commits into
TheSmokeDev:mainfrom
kvnloo:feat/realtime-semantic-endpoint-adapter
Open

kvnloo wants to merge 2 commits into
TheSmokeDev:mainfrom
kvnloo:feat/realtime-semantic-endpoint-adapter

Conversation

@kvnloo

@kvnloo kvnloo commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What and why

Mirror the provider-neutral turn-detection contract in Talk's wire boundary: map OpenAI server/semantic VAD through mint and live update paths, advertise Grok native/server support and fail fast on semantic mode, advertise Gemini native-only support and fail fast on server/semantic modes. The semantic turn-detection names are probed as an optional capability, so pre-semantic core heads (today's #101808) keep the full core lane with provider-native turn detection instead of losing it. Ships a deterministic, content-free endpointing benchmark module.

Pairs with TheSmokeDev/hermes-agent#4 (stacked on NousResearch/hermes-agent#101808).

How to test

  • Full suite: 1580 passed, 56 skipped, 5 xfailed
  • tests/test_core_provider.py: 8 passed (was 7), 38 skipped without a core checkout
  • Ruff passes on every changed Python file
  • Wheel build succeeds and includes talk_endpointing_benchmark.py
  • Benchmark smoke reports endpoint p50/p95/max, playback latency, timeout, cutoff/split, and false activations without retaining audio or transcripts

Platforms

  • Linux (CI + developer box)
  • Windows/macOS: untouched paths only; no platform-specific code in this PR

Live receipt

No live-provider run in this PR: provider lanes are exercised through contract-shaped unit coverage and the benchmark smoke module. Live-lane receipt stays with hermes talk check (#97) on a configured box.

Checklist

  • Failing behavior reproduced first (old head dropped the whole core lane with a bare ImportError)
  • Regression test binds the #101808-shaped contract and fails without the fix
  • Full suite green (1580 passed, 56 skipped, 5 xfailed)
  • Ruff clean on every changed file
  • CHANGELOG carries the endpointing bridge entry, not just the benchmark
  • No credentials, transcripts, or audio retained in tests or artifacts

@TheSmokeDev TheSmokeDev left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kevin — this is a clean design: neutral RealtimeTurnDetection, exact GA semantic_vad shape, and honest Grok/Gemini refusals. One blocker before I can take it: talk_core_provider hard-imports RealtimeSemanticEagerness / RealtimeTurnDetection / RealtimeTurnDetectionMode inside the contract try, so on today's #101808 head (without your hermes-agent#4) the whole core lane disappears — core_contract_available() goes False with a bare ImportError, and the widened _REAL_CONTRACT_MARKERS makes every contract test skip instead of fail. Could you probe those three names as an optional capability (own try/except, getattr on validate_setup / setup.turn_detection, default to native), and add a test that binds the #101808-shaped contract? Also a CHANGELOG line for the endpointing bridge itself, not just the benchmark. Happy to merge right after. — SmokeDev

@kvnloo

kvnloo commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 412c673 (full suite 1580 passed, 56 skipped, 5 xfailed, ruff clean): the three semantic names are now imported in their own try/except with a turn_detection_available() probe, mode sets degrade through _contract_turn_modes, and a missing setup.turn_detection defaults to provider-native via getattr. test_old_head_contract_keeps_core_lane_with_native_only_turn_detection binds a synthetic #101808-shaped contract (API v2, no turn-detection names) and proves the lane stays up degraded with native-only advertisement and a clean ValueError on semantic. Also added the bridge CHANGELOG entry and remapped the body to the repo template.

- Import RealtimeSemanticEagerness/RealtimeTurnDetection/
  RealtimeTurnDetectionMode in their own try/except so a pre-semantic
  core head keeps the full lane instead of dropping it on ImportError
- Default missing setup.turn_detection to provider-native via getattr;
  refuse non-native modes before resources open
- Advertise native-only mode sets on old heads via _contract_turn_modes
- test_old_head_contract binds a #101808-shaped contract and proves
  degraded-mode behavior
- CHANGELOG: endpointing bridge entry
@kvnloo
kvnloo force-pushed the feat/realtime-semantic-endpoint-adapter branch from 412c673 to 5c93c18 Compare September 10, 2026 04:13
kvnloo pushed a commit to kvnloo/hermes-talk that referenced this pull request Sep 10, 2026
…, first issues

Three outside contributors showed up in one day (kvnloo, TheAngryPit,
webdevtodayjason) and found a CONTRIBUTING that covered setup and lint but
not what we want, where a change starts, or what a PR has to carry. This
makes the next thirty obvious. Markdown and YAML only; no Python changes.

CONTRIBUTING.md, rewritten around the existing facts (every one kept —
setup, the ruff pin, the TheSmokeDev#93 baseline, the module table now including
talk_check and talk_diagnostics, the adversarial-review paragraph, the
SHA-pinned-actions and BLE001/RUF100 rules):
- ranked priorities for THIS repo: live-lane bug fixes > provider/host
  compatibility > security hardening (auth store, token routing, redaction)
  > cross-platform > new providers behind the contract > new surfaces > docs
- a common-paths table — provider, surface, talk tool, fix, docs — naming
  the files each touches and ships with (talk_realtime.py contract,
  talk_core_provider.py lane, talk_config/talk_cli/talk_doctor wiring,
  pyproject py-modules; the DuplexAudio method set a surface implements;
  the _HANDLERS + READ_ONLY/MUTATING classification a tool needs)
- why `uv run --extra dev`, node for the two dashboard JS tests, and how to
  run a branch live (`hermes plugins install --force --ref <sha>`)
- branch names and Conventional-Commit scopes derived from the last 200
  commits, the PR description contract, the merge bar, a
  first-response-within-24h review promise, and how credit works

.github/PULL_REQUEST_TEMPLATE.md carries the same contract: what/why, how
to test, platforms, a `check --json` receipt when a lane is touched,
`Fixes #N`, a checklist that repeats the merge bar.

Issue forms: feature_request.yml (problem first, proposal second, which
surfaces and providers), provider_compatibility_report.yml (PASS/PARTIAL/
FAIL, provider, model, credential lane, versions, which events were
observed — SessionReady / SpeechStarted / FunctionCall round-trip /
barge-in — the `check --json` report, the wire error verbatim; no audio,
transcripts, or secrets), bug_report.yml gains the core-contract lane and
the check/doctor/diagnostics commands as surfaces plus a pointer to the
provider form, config.yml links private security reporting and
Discussions first.

docs/PROVIDER-RECEIPT.md: the two-minute procedure (`check --json`, then
one short call with a five-row tick list), the table the form mirrors,
and how maintainers act on each verdict — the unticked event names the
adapter function; the wire error becomes the regression fixture. Ported
idea from bielcarpi/hermes-live-voice's compatibility receipt (MIT) —
idea only, no text.

README: Contributing shortened to the pointer plus the promise, a
Contributors line crediting @kvnloo (TheSmokeDev#81 merged, TheSmokeDev#107 in review),
@TheAngryPit (TheSmokeDev#80 in review), @webdevtodayjason (second-consumer feedback
on upstream hermes-agent#81404, the contract our lanes register on).
CHANGELOG [Unreleased] → Added.

Repo settings alongside (not in this diff): labels provider, surface,
security, docs created; good first issue + help wanted on TheSmokeDev#93, help wanted
on TheSmokeDev#90 and TheSmokeDev#51; Discussions were already enabled.

— SmokeDev
@kvnloo

kvnloo commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (conflicts cleared; only CHANGELOG.md needed a minimal resolve). Prior CR already addressed at 412c673 / now 5c93c18 (optional semantic turn-detection probe + test_old_head_contract + CHANGELOG endpointing bridge). Please re-review when you have a moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants