synaptic

# GPT-Live voice sessions

GPT-Live 1 is an opt-in OpenAI voice integration. Existing Realtime, duplex,
turn-based, and Gemini callers keep their current defaults and event contracts.
Selecting `gpt-live-1` alone on a Realtime session is not enough: choose the Live
experience explicitly. This implementation supports **WebRTC with client
delegation**. Responses delegation, primary audio WebSockets, and a provider
sideband WebSocket are not implemented here.

## Connect a browser

Have the browser create a peer connection, attach microphone tracks, register an
`oai-events` data channel, create an SDP offer, and finish ICE gathering. Send the
offer to your application server. Keep the OpenAI API key on that server.

```elixir
{:ok, session} = Synaptic.Voice.start_session(MyApp.ConversationWorkflow, %{},
  mode: :realtime,
  provider: :openai,
  experience: :live,
  profile: MyApp.VoiceProfile,
  session_context: %{preferred_language: "en"},
  provider_opts: [realtime: [
    model: "gpt-live-1",
    voice: "marin",
    sdp: browser_offer
  ]]
)

Synaptic.Voice.subscribe_session(session.session_id)
# Return session.transport.sdp to the browser as its remote SDP answer.
```

The profile declares the fields accepted by `session_context`; omit both options
to use the default profile. Bootstrap calls `POST /v1/live/sessions` and requires
HTTP 201. It does not retry this billable creation request. `transport.session_id`
is OpenAI's opaque Live ID; `session.session_id` is Synaptic's local session ID.

Wait for `session.started`, not merely the data channel opening. Do not send
Realtime `session.update`, `response.create`, audio commits, or `session.start`.
WebRTC media tracks carry audio. Forward JSON provider events with
`Synaptic.Voice.ingest_provider_event/2`. Forward the `data.event` from each
`:provider_outbound` envelope back onto the data channel exactly once, in order.
Subscribe and attach the browser to its application session **before** applying
the SDP answer so early events cannot race the subscription.

As with the existing lab, this browser relay is a development integration. A
production host must authenticate sessions and enforce ownership of inbound
events; do not treat browser-supplied provider events as proof of authorization.

## Connect the backend

The default backend resumes the attached Synaptic workflow at a human checkpoint
with `%{human_input_text: role_labelled_conversation}`. It waits for the next human
checkpoint or completion and reads `:assistant_answer` (or `:answer`, `:response`,
`:reply`). Workflows remain responsible for their business rules, tool permissions,
confirmations, and operation idempotency.

Live delegation is not a structured function call. It supplies an opaque ID, not
task text or tool arguments. Synaptic retains a rolling window of 2,000 transcript
fragments, preserving exact text and timing. It waits for user context and a short
settling interval (350 ms by default). This delay is only a delivery heuristic,
not proof of a complete sentence. Use `live_context_fun` for stronger
application-specific readiness checks; subscribe to transcript events if your
application needs the full history or persistence.

To connect a different agent or an application backend:

```elixir
live_delegate_fun: fn snapshot ->
  # snapshot includes run_id, session_id, delegation_id, revision, transcript,
  # timeout_ms, session_context, authorized capabilities, and security_policy.
  MyApp.VoiceBackend.run(snapshot) # => {:ok, concise_verified_text} | {:error, reason}
end,
live_context_fun: fn snapshot ->
  # A fast, non-blocking callback. A subsequent user fragment retries :wait.
  if MyApp.RequestContext.ready?(snapshot), do: {:ok, snapshot}, else: :wait
end
```

Profile capabilities are filtered using existing session authorization. Their
descriptions are available to the voice model and their definitions to the
backend callback. They are **not** executed automatically by the Live engine.
An application backend selecting direct capabilities should call
`Synaptic.Voice.CapabilityGateway.execute/5` and preserve application confirmation
checks. `approve_capability/2` is not supported by this engine: backend
confirmation is separate from the voice stream.

Backend output must be 1–500 UTF-8 bytes. This conservative bound fits the API's
500-token append limit without adding a tokenizer. Longer results return
`:live_result_too_long`; they are never silently truncated. Summarize inside the
backend before returning. Output is sent as `session.commentary.append`; actual
spoken text comes from the provider's output transcript, not this result.

A backend with a stable committed-result identity may instead return
`{:ok, %{text: concise_verified_text, event_id: stable_id}}`. The text keeps the
same 1–500 UTF-8 byte limit; `event_id` is a server-owned string of 1–128 bytes.
Live emits that ID on the commentary append and remembers up to 256 identified
replies for this voice session. Repeating the same ID and text finishes the
backend delegation without another release. Reusing an ID with changed text
returns `:delivery_identity_conflict`; exceeding the retained identity limit
returns `:delivery_capacity`. Existing binary results retain their prior behavior.

This contract deduplicates release emission. It does not prove provider acceptance
or playback, persist across session loss, or authorize a business effect. Hosts
must correlate provider acknowledgement IDs separately and must not replay an
uncertain audio delivery automatically. A browser relay should also deduplicate
identical event IDs and reject conflicting payloads.

## Corrections, failures, and lifecycle

- Delegation IDs and provider event IDs suppress duplicate delivery. Only one
  backend task runs at a time; up to 32 delegation notices can wait.
- New user transcript text increments a revision. If it changes while a backend
  runs, the result is withheld and `:delegation_superseded` is emitted. This is
  deliberately conservative: even a short acknowledgment may invalidate an
  answer. It does not automatically rerun or reverse an action.
- Backend tasks are supervised without linking their failures to the voice
  process. A timeout or crash blocks further execution for that session, since
  the operation's outcome may be unknown. Reconcile application state before
  creating a replacement session. Stopping a waiter does not cancel a workflow.
- User and assistant speech may overlap. `:input_transcript_delta` and
  `:assistant_text_chunk` carry exact fragments and timestamps. There is no fake
  `:input_final_text` or `:assistant_response_done` event. Keep playback status
  separate from transcript and workflow progress.
- `stop_session/2` requests `session.close` and enters `:closing`. Keep receiving
  and forwarding events until `session.closed`. `:session_usage` has cumulative
  usage and `final: true` only after that final event. The engine then stops.
- A lost connection or a 15-second close timeout stops the session with final
  usage unconfirmed. Workflow state is independent and in-memory; it is not
  persisted across application restarts. Automatic reconnect/replay is not used.
- `push_audio`, `push_text`, `end_turn`, and `cancel_output` are not supported for
  this WebRTC integration. Microphone and playback controls belong in the client.

## Voice lab

The local demo is `tmp/voice_lab` (underscore), and remains gitignored. It uses
the repository through a path dependency. Start it with `OPENAI_API_KEY` set:

```sh
cd tmp/voice_lab
mix assets.build
VOICE_LAB_PORT=4011 mix phx.server
```

Open `http://localhost:4011`, select **OpenAI → Realtime → GPT-Live 1**, then
connect. Realtime 2.1 and mini remain available for comparison. Live uses Marin;
the existing Realtime demo uses Verse. The interview profile is preserved, with
no application tools enabled. If reasoning is delegated, `VoiceLab.Workflow`
remains the backend. The UI hides Realtime-only reasoning and conversation mode
controls when Live is selected.

Try interrupting the greeting, overlapping speech, a multi-part answer, a
correction while backend work runs, disconnecting, and reconnecting. Compare
both spoken behavior and transcript/event logs. Mocked tests validate protocol
and lifecycle behavior; real model access, microphone behavior, and spoken
quality require this manual test.

Official references: [Live WebRTC](https://developers.openai.com/api/docs/guides/voice-webrtc?api=live),
[client delegation](https://developers.openai.com/api/docs/guides/live-delegation),
[session lifecycle](https://developers.openai.com/api/docs/guides/live-conversations).