Skip to content

API.md

Latest commit

 

History

History
192 lines (153 loc) · 12.7 KB

API.md

File metadata and controls

192 lines (153 loc) · 12.7 KB
Raw

Earshot API

mod_earshot registers a dialplan app and an API named earshot (plus a uuid_audio_stream/audio_stream compat shim), a set of earshot:: custom events, and a few EARSHOT_* channel variables. Options are named (key=value) — no positional order to memorize, and new options never shift an existing one.

  • App (dialplan, current channel): <action application="earshot" data="<verb> …"/>
  • API (fs_cli / ESL, explicit uuid first): earshot <uuid> <verb> …

Verbs

start <url> [key=value …]

Begin streaming the channel's audio to url (ws:// or wss://).

Option Values Default Notes
id <name> (default stream) fan-out: name this stream so many can run on one channel
proto native twilio openai deepgram vapi elevenlabs gemini pipecat assemblyai cartesia native wire adapter (see Protocol adapters)
codec pcmu pcma l16 pcmu g711 = 8-bit telephony (½ the bytes); some protos pin the codec
rate 8000 16000 24000 8000 wire rate; resampled to/from the channel rate
dir in out both both in = caller→agent only (read-only fork, no playback); both = bidirectional agent
ready firstframe connect manual firstframe when playback opens (see Ready gate)
vad on off module-side VAD → speech_started/speech_stopped
vad_barge on off back-compat alias for interruptible=speech
interruptible none dtmf speech any none (or speech if vad_barge=on) what interrupts the agent's playback
ignore_backchannel on off require sustained speech before a speech-barge (drops short "yeah/okay")
sensitivity low medium high — preset for the sustained-speech gate (600 / 300 / 150 ms)
barge_min_ms <n> 0 ms of sustained caller speech before a speech-barge (0 = immediate)
barge_fade_ms <n> 0 fade playback out over N ms instead of a hard cut
vad_notify on off also send {"type":"speech_started"} to the agent
vad_mode / vad_voice_ms / vad_silence_ms ints 2 / 200 / 500 VAD aggressiveness + endpointing budget
dtmf on off capture caller DTMF → earshot::dtmf + forward to the agent
mask on off start in a PCI masking window (mute audio + redact DTMF to the agent)
commands true false opt-in control channel (agent can drive the call)
metrics <seconds> 0 emit earshot::metrics every N seconds
corr auto | <id> auto correlation id; auto = SIP Call-ID
auth <token> (no spaces) — Authorization header. A value containing a space (Bearer <key>, Token <key>) must be passed via the EARSHOT_AUTH channel variable instead — the option list is split on spaces, so auth=Bearer … would truncate to Bearer
greeting <file> (no spaces) — welcome audio played into the channel the moment the ready gate opens, ahead of the agent's first words (see Welcome greeting). A path with spaces goes via the EARSHOT_GREETING channel variable

Caller context (customer id, tier, call reason, …) is passed to the agent at connect via the EARSHOT_META channel variable — see Caller context.

stop · pause · resume

Tear down / suspend / resume the stream. pause freezes both directions: caller audio and DTMF stop going to the agent, queued agent audio is held (not dropped) and barge-in is suspended until resume, which also opens a ready=manual gate. FreeSWITCH's own audio (ringback, hold music) keeps flowing while paused. Channel hangup also tears down cleanly.

flush — barge-in

Immediately drop queued agent audio (also driven by protocol clear signals and vad_barge).

send <text|json>

Write a raw message to the agent socket.

mask on|off

Toggle the PCI masking window on every stream of the channel (fan-out streams included); operator control, and the agent can also toggle it via the control channel.

status · metrics

Return per-stream JSON. metrics also fires an earshot::metrics event. Fields include proto, corr, id, tx/rx_frames, tx/rx_bytes, play_drops, commands, speech_starts, talking, dtmf, masking, ws_connected, ws_reconnects, ws_queue_drops, and the latency KPIs first_audio_ms, response_ms / response_ms_max, turns, ws_rtt_ms.

Targeting a fan-out stream: every non-start verb accepts an id=<name> immediately after the verb, e.g. earshot <uuid> status id=transcribe, earshot <uuid> stop id=supervisor.

Control channel (commands=true)

The agent sends {"type":"command","action":…} over the same socket; Earshot maps each whitelisted action to a thread-safe uuid_* API, audits it via earshot::command, and replies command_result.

{"type":"command","action":"transfer","to":"2000","id":"c1"}   // -> uuid_transfer
{"type":"command","action":"send_dtmf","digits":"1"}
{"type":"command","action":"mask","state":"on"}                // PCI masking

Actions: transfer · hangup · send_dtmf · play · stop_play · record · setvar · hold · bridge · park · mask. Off by default; unknown/disabled → {"ok":false,"error":…}.

Events (subclass earshot::…)

Event Key headers Fires when
earshot::connected url WS handshake completes
earshot::ready corr playback gate opens (also sets earshot_ready=true)
earshot::speech_started / earshot::speech_stopped corr VAD turn boundaries (sets earshot_talking)
earshot::dtmf digit or masked caller DTMF (masked=true, digit redacted, during a mask window)
earshot::transcript text, final, corr, stream-id STT result (proto=assemblyai); final=true is an end-of-turn result, false a partial
earshot::command action, api, ok, result a control-channel command ran
earshot::metrics counters + latency KPIs (first-audio-ms, ws-rtt-ms, …) periodic / on-close / on-demand

All subclasses are reserved, so ESL subscribers receive them: event plain CUSTOM earshot::metrics.

Ready gate

Playback into the channel is held until media is confirmed flowing, so an agent's first words never land in silence. ready=firstframe (default) opens on the agent's first audio frame; ready=connect on the WS handshake; ready=manual waits for earshot <uuid> resume. Opening fires earshot::ready and sets the earshot_ready channel variable.

Welcome greeting

greeting=<file> (or the EARSHOT_GREETING channel variable for paths with spaces) plays an audio file to the caller the instant the ready gate opens, ahead of the agent's first words — a module-owned prompt that never lands in silence and needs no round-trip to the model. The file is loaded once at start and resampled to the channel rate (any format FreeSWITCH can open; bounded to 15 s). It is emitted into the playout before any agent audio — whichever thread opens the gate emits it first — so it leads in every ready mode (firstframe, connect, manual), with the agent's reply queued behind it. Use ready=connect when you want the greeting to play as soon as the socket connects rather than waiting on the agent's first frame. Barge-in (flush / vad_barge) cuts it like any other playout. Read-only forks (dir=in) never play back and ignore it.

For a dynamic greeting, prefer the agent's own (e.g. Deepgram's greeting in EARSHOT_SESSION_CONFIG) or pre-render the text to a file — greeting= is a fixed audio prompt, not TTS.

Caller context

Set the EARSHOT_META channel variable to pass per-call context to the agent at setup — a customer id, account tier, call reason, campaign, anything. Earshot sends it verbatim as the X-Earshot-Meta header on the WebSocket handshake (alongside X-Call-ID / X-Channel-UUID / X-Correlation-ID), so it's readable by any agent framework at connect, regardless of proto. Keep it compact and single-line (it's an HTTP header) — JSON is the natural shape:

<action application="set" data="EARSHOT_META={&quot;customer_id&quot;:&quot;C-8842&quot;,&quot;tier&quot;:&quot;gold&quot;,&quot;reason&quot;:&quot;billing&quot;}"/>

This is transport-level context for the handshake; proto-specific in-band session config (voices, models, prompts) still goes through EARSHOT_SESSION_CONFIG.

Protocol adapters

proto= selects how audio + control map onto the wire, so an existing agent works unchanged. Full framing details in FEATURES.md; in brief:

  • native — raw binary frames + {"type":"playAudio"|"clear"} JSON control.
  • twilio — Twilio Media Streams (connected/start/media/mark, base64 µ-law, mark echo).
  • openai — OpenAI Realtime (session.update, input_audio_buffer.append, response.output_audio.delta; realtime subprotocol, auth via EARSHOT_AUTH).
  • deepgram — Deepgram Voice Agent (Settings, raw binary audio, UserStartedSpeaking).
  • vapi — Vapi WebSocket transport (raw binary audio, speech-update/user-interrupted control). The operator does a REST POST /call with transport.provider="vapi.websocket" and passes the returned per-call websocketCallUrl as the start url; match codec=/rate= to the call's audioFormat (e.g. codec=l16 rate=16000 ↔ pcm_s16le/16000, or codec=pcmu rate=8000 ↔ mulaw/8000).
  • elevenlabs — ElevenLabs Conversational AI (user_audio_chunk / {type:audio}, auto ping→pong).
  • gemini — Gemini Live (setup, realtimeInput.mediaChunks 16k / serverContent 24k; resampled).
  • pipecat — Pipecat protobuf Frame{ audio: AudioRawFrame } (binary L16).
  • assemblyai — AssemblyAI Universal-Streaming STT (wss://streaming.assemblyai.com/v3/ws?sample_rate=…). Transcription only: PCM16 audio out (codec=l16, coalesced to ≥50 ms chunks), transcript JSON in → earshot::transcript events. Use dir=in as a read-only fork; auth is the raw API key in the Authorization header via EARSHOT_AUTH. No audio comes back (no playback).
  • cartesia — Cartesia ink-whisper STT (wss://api.cartesia.ai/stt/websocket?model=ink-whisper&…). Transcription only: PCM16 audio out (codec=l16, ~100 ms chunks), {"type":"transcript",…} in → earshot::transcript. Use dir=in. The raw API key rides the URL as access_token= (with encoding=pcm_s16le, sample_rate=, cartesia_version=) — no auth header needed.

For openai/deepgram/gemini/elevenlabs, provide the full session config (voice/model/keys) via the EARSHOT_SESSION_CONFIG channel variable; otherwise Earshot sends an audio-format default. For vapi the assistant/voice/model are set in the POST /call body, not over the socket.

Channel variables

Variable Purpose
EARSHOT_SESSION_CONFIG verbatim first message for openai/deepgram/gemini/elevenlabs
EARSHOT_META opaque caller context → X-Earshot-Meta handshake header (see Caller context)
EARSHOT_GREETING welcome-audio file path (alternative to greeting= for paths with spaces)
EARSHOT_NO_RECONNECT disable auto-reconnect (default: reconnect with jittered backoff)
EARSHOT_TLS_NO_HOSTNAME_CHECK skip wss cert/hostname checks (dev only)
earshot_ready / earshot_talking / earshot_masking set by Earshot for dialplan logic

Box-level TLS (environment)

Mutual TLS and custom-CA verification are configured per box (not per stream), via process environment variables read once at module load and applied to every agent connection:

Env var Purpose
EARSHOT_TLS_CLIENT_CERT PEM client certificate to present (enables mTLS)
EARSHOT_TLS_CLIENT_KEY matching private key (unencrypted PEM)
EARSHOT_TLS_CA verify the agent against this private CA instead of the system trust store — applies to every connection on the box, so public-CA endpoints (OpenAI, Deepgram) will fail; set only when all agents chain to this CA

One client identity per box — libwebsockets binds client TLS material at the shared-context level, so distinct per-stream certificates are not supported.

Compatibility

uuid_audio_stream / audio_stream accept mod_audio_stream's positional syntax (start <url> <mix> <rate>) and run existing dialplans unchanged — registered only when those names are free (won't collide with a loaded mod_audio_stream). See FEATURES.md.