Voice agents

6.0 What a voice agent is

A voice agent is a conversation partner in realtime. You open one WebSocket and stream the audio of the person speaking: a caller on a telephone line, or a user with a microphone in a browser or app. The agent listens, understands and answers with natural speech, following the instructions you give it. You build the phone bot or voice assistant around it; the listening, understanding and speaking run on this one stream.

What you get:

• Swiss dialects and regional varieties, both ways. The agent understands and speaks Swiss German dialects (Zürich, Bern, Basel, Lucerne and general Swiss German), Swiss French (fr-CH) and Swiss Italian (it-CH), and also Standard German, French, Italian and English.
• Your own instructions: persona, knowledge and rules, and the names and terms it should recognise.
• Tools: the agent asks your backend to look something up or book something, and speaks the result.
• Natural interruptions (barge-in), telephone audio (G.711 µ-law and A-law at 8 kHz) and PCM at 8, 16, 24 and 48 kHz.
• Call control: the agent can end a conversation on the caller’s goodbye, and hand a caller over to a person (6.5).
• Live transcripts of both sides, and the whole conversation in session.complete.
• Browser-safe access: short-lived single-use tokens, so a page never holds your API key (6.1).

Voice agents are billed per second of connected session time, at CHF 0.33 per minute (CHF 19.80 per hour), excluding VAT, from the same prepaid credit as the other services (6.7). Sandbox keys run a synthetic session and are never billed.

Use speech-to-text and text-to-speech directly (5, 3) when you want to run the conversation logic yourself; use a voice agent when you want the conversation itself.

6.1 Connect: GET /agent/stream (WebSocket)

Endpoint: wss://api.suisse-speech.ch/v1/agent/stream

Authentication

Authenticate like the other sockets (Authentication):

A browser cannot set handshake headers, so it passes the credential in the subprotocol; list suisse.v1 as well, it is the subprotocol the server answers with. The query parameter ?api_key= is accepted too, but avoid it: query strings end up in logs. Never put your API key into code that runs in a browser or an app you ship: your server mints a token, the browser connects with it.

A handshake that is refused has no body, so it answers with the HTTP status alone: 401 (no valid key, or a token that is unknown, already used or expired), 429 (no capacity, the plan’s simultaneous sessions, or the session rate; read Retry-After and X-Suisse-Capacity-Reason) or 503 (tokens are unavailable just now). Voice agents are released per account: for an account without them the socket opens, sends an error with agent.not_enabled and closes with 1008. If agent.not_enabled comes back, contact us and we enable voice agents for your account. GET /capabilities says whether your key may open a session (agent.enabled_for_your_account).

Browser tokens: POST /agent/tokens

Your server asks for a token with its API key and hands it to the page. The token opens one session and is worth nothing afterwards.

curl -X POST https://api.suisse-speech.ch/v1/agent/tokens \
  -H "X-API-Key: $SUISSE_SPEECH_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"lock": true, "session": {"language": "de-CH", "dialect": "zurich",
       "voice": "anna", "instructions": "Du bist die Rezeption der Garage Muster."}}'

{"token":"sat_Q2xhc3NpY0Jsb2NrU3RyZWFtMTIzNDU2","expires_at":"2026-11-03T09:01:00.000Z","expires_in":60}

• A token is sat_ followed by 32 characters. It is valid for 60 seconds and for one handshake: it is used up by the first connection attempt, even one that is then refused with 429. Mint a new token for every session.
• It belongs to the account and the environment of the key that minted it (a sandbox key mints a sandbox token). If that key is revoked, the token stops working.
• The body is optional, and strict: JSON only (Content-Type: application/json), no query parameters, and only session and lock at the top level. session presets any session.start fields; it is checked with the rules of session.start when the token is minted, so an unknown field, a wrong type, or an unknown voice, language or dialect is refused at once with 400 agent.invalid_request, not later in the browser. lock (default false) fixes some of them.
• Merging. The browser’s own session.start is applied over the preset, field by field: a field the browser sends replaces the preset’s field as a whole, every other preset field stays.
• Lock. With "lock": true the preset’s instructions, tools, voice, language, dialect and greeting are fixed; one the preset leaves out stays unset, at its default. A session.start that sends one of them with another value, or one the preset leaves out, is refused with agent.forbidden and the socket closes with 1008; sending exactly the preset’s value is fine. During the session, config.update and reply with instructions are answered with agent.forbidden and ignored; the session goes on. Use lock together with session: it is what keeps the page from rewriting your agent.
• Errors: 400 agent.invalid_request, 401 agent.unauthenticated, 403 agent.not_enabled, 429 agent.rate_limited, 503 agent.unavailable.

A secure browser flow, in two small pieces. Your server mints the token and should only do so for a visitor you have authenticated yourself:

// Node.js (Express); `requireLogin` is your own check
app.post('/agent-token', requireLogin, async (req, res) => {
  const r = await fetch('https://api.suisse-speech.ch/v1/agent/tokens', {
    method: 'POST',
    headers: { 'X-API-Key': process.env.SUISSE_SPEECH_KEY, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      lock: true,
      session: {
        language: 'de-CH', dialect: 'zurich', voice: 'anna',
        instructions: 'Du bist die Rezeption der Garage Muster. Antworte kurz.',
        greeting: 'Grüezi, Garage Muster. Wie kann ich helfen?',
        tools: [{ name: 'get_time', description: 'Returns the current time.',
                  parameters: { type: 'object', properties: {} } }],
      },
    }),
  });
  res.status(r.status).json(await r.json());
});

The browser fetches a token and connects with it. The agent’s character is locked in the token, so the page sends only what is its own, the audio formats:

const { token } = await (await fetch('/agent-token', { method: 'POST' })).json();
const ws = new WebSocket('wss://api.suisse-speech.ch/v1/agent/stream', ['suisse.v1', 'key.' + token]);
ws.onopen = () => ws.send(JSON.stringify({
  type: 'session.start',
  input:  { encoding: 'pcm16', sample_rate_hz: 16000 },
  output: { encoding: 'pcm16', sample_rate_hz: 24000, transport: 'binary' },
}));

session.start

The first message must be session.start, and it is sent once, within 10 seconds of connecting. Control messages are JSON text frames with a type field. Every field is optional.

{
  "type": "session.start",
  "language": "de-CH",
  "dialect": "zurich",
  "voice": "anna",
  "instructions": "Du bist die Rezeption der Garage Muster. Antworte kurz und freundlich.",
  "greeting": "Grüezi, Garage Muster. Wie kann ich helfen?",
  "input":  { "encoding": "pcm16", "sample_rate_hz": 16000 },
  "output": { "encoding": "pcm16", "sample_rate_hz": 24000, "transport": "binary" },
  "turn_taking": { "mode": "auto", "interruptions": true, "patience": "normal" },
  "vocabulary": ["Muster", "Bremgarten"],
  "pronunciations": [{ "text": "SIT", "say": "Suisse I T" }],
  "max_duration_s": 600
}

turn_taking: mode auto (default) or manual; interruptions true (default) or false; patience short (default), normal or long; idle_prompt_s, a whole number from 5 to 60, default 10; idle_end_s, a whole number from 15 to 300, default 45, and longer than idle_prompt_s. The two idle settings apply in auto mode only (6.5).

instructions and tools together take at most 24,000 characters. A field that is not listed is refused with agent.invalid_request, and so is a value outside its range; a typo never silently does nothing. detail names the position of the problem, never the value you sent. A session.start that is refused ends the connection with 1008.

The server answers with session.ready once the session is set up:

{"type":"session.ready","request_id":"req_…","session_id":"req_…","voice":"anna",
 "language":"de-CH","dialect":"zurich",
 "input":{"encoding":"pcm16","sample_rate_hz":16000},
 "output":{"encoding":"pcm16","sample_rate_hz":24000,"transport":"binary","frame_ms":40,"frame_bytes":1920},
 "turn_taking":{"mode":"auto","interruptions":true,"patience":"normal","idle_prompt_s":10,"idle_end_s":45},
 "languages":["de-CH"],"speed":1,"end_conversation":true,"transfer_targets":0,"pronunciations":1,
 "tools":0,"transcripts":true,"max_duration_s":600,"limited_by":"max_duration_s"}

session.ready repeats the settings the session runs with. max_duration_s is the length this session can really have: if the account’s remaining credit is shorter than the max_duration_s you asked for, it is shortened and limited_by is credit (otherwise max_duration_s). Billing starts at session.ready (6.7). The greeting, if you set one, is spoken straight away.

6.2 Audio in and out

Sending audio. Send the audio in the encoding and rate you declared in input, either as binary frames (raw samples, nothing else; at most 64 KB each) or as JSON:

{"type":"audio.append","data":"<base64>"}

Send audio at real-time pace, from session.ready on. The service accepts at most 1.2 times real time (with a burst of 2 seconds); audio beyond that is dropped and you get one error with agent.invalid_audio; the session goes on. You may send at most 100 frames per second, text and binary together; the excess is dropped and you get agent.rate_limited, at most once per second. Frames of 20 to 40 ms stay well within it. data that is not valid base64 is answered with the same error. Binary frames are the better choice: no base64, no parsing.

Receiving audio. The agent’s voice arrives in frames of 40 ms each (frame_ms and frame_bytes in session.ready; the last frame of a reply may be shorter), in order. With "transport":"binary" in output they are binary frames between reply.started and reply.audio_done; a binary frame is always audio. With "transport":"json" the same audio arrives as audio.chunk messages (reply_id, seq counting from 1, base64 data). After audio.clear, no frame of that reply follows.

Pacing. The agent’s audio leaves on the service’s own clock: the first 200 ms or so of a reply at once, then a frame every 40 ms at about 1.25 times real time, and never more than about 1.5 seconds ahead of your playback. Your playback buffer therefore stays small, and audio.clear takes effect at once. The events of a reply keep their order with its audio: reply.audio_done and reply.done come after its last frame, and the next reply’s reply.started after the previous reply.done. Transcript deltas keep their order with the audio too, so a reply.transcript.delta can arrive slightly after the audio it belongs to; do not use it to time playback. Deltas that wait behind audio may arrive merged into one.

A reply’s audio is checked against its own transcript before it is sent, so the first frame of a reply comes a moment after its first words are ready. While more than about 1.5 seconds of audio still waits to go out, the next reply starts only once it has gone out; nothing is lost.

Formats. pcm16 is headerless 16-bit little-endian mono PCM at 8000, 16000, 24000 or 48000 Hz. mulaw and alaw are G.711 at 8000 Hz, the format of telephone lines, so call audio can pass straight through without conversion. Input and output formats are independent: a browser can send 16 kHz PCM and play 24 kHz PCM, a telephone gateway can use µ-law at 8 kHz both ways.

A client that reads too slowly, or a very long reply. If your connection falls more than about two seconds of audio behind, the excess audio is dropped and you get a session.warning with agent.audio_dropped and the reply_id; the next reply is sent normally. If a single reply runs more than about 30 seconds ahead of your playback, you get the same warning, then audio.clear (only if audio of that reply already reached you) and reply.done with status interrupted. The conversation continues from what the listener actually heard.

Echo cancellation is required on your side. The agent listens while it speaks; if your client plays the reply through a loudspeaker and the microphone picks it up again, the agent hears itself and interrupts itself. Telephone lines cancel echo already. In a browser, ask for it: getUserMedia({ audio: { echoCancellation: true } }). A headset is the safest setup, and worth recommending to your users wherever they talk for longer than a moment.

6.3 The conversation

Events from the server

On a sandbox key, every event carries "sandbox": true.

With transcripts true (the default), user.transcript.* and reply.transcript.delta are sent while the conversation runs, reply.done carries the reply’s transcript, and session.complete carries the whole transcript. With false, none of these are sent: reply.done has no transcript either. user.speech_started and user.speech_stopped come either way.

Turn-taking. In mode: "auto" the agent decides when the caller has finished and then answers; patience sets how long it waits after a pause (short, normal, long). Use long for callers who think aloud, short for quick exchanges. In mode: "manual" the agent answers only when you send input.commit, for example from push-to-talk; in auto mode input.commit is refused with agent.invalid_request.

Interruptions. With interruptions: true, a caller who speaks while the agent is talking interrupts it: two words, a stop word such as “stopp” or “moment”, or longer speech; a short “mhm” does not. The agent stops, and the server sends audio.clear for the reply that was playing. Drop everything you have buffered for that reply_id immediately, otherwise the caller hears the agent talk on for as long as your buffer is long. The reply ends with reply.done and status interrupted; its transcript is what the agent had said. You can also stop the agent yourself with interrupt.

playback.mark: tell the agent what was heard. The server knows how much audio it has sent, not how much of it your client has played. Audio waits in your buffer and in the network, so on a slow or jittery connection the caller is often several sentences behind. When the caller interrupts, the agent has to know where its sentence was cut: otherwise it believes the caller heard words they never heard and carries on from the wrong place. Send playback.mark (reply_id, played_ms) about every 250 ms while a reply plays:

{"type":"playback.mark","reply_id":"rp_…","played_ms":1840}

played_ms is how many milliseconds of that reply your client has played, counted from the reply’s first audio. A mark counts for its own reply only. It must not decrease and must not exceed the audio sent for the reply, otherwise it is answered with an error (agent.invalid_request) and ignored. A mark for a reply that no longer plays (interrupted, cleared by audio.clear, or long finished) is accepted and ignored, never an error, so a final mark after audio.clear is fine. Without marks the server estimates from the clock, which is accurate only while your playback keeps pace. It is optional, and worth it on browser and mobile clients.

6.4 Tools

Tools let the agent act on your systems. Define them in session.start (or later with config.update):

{"tools":[{
  "name": "book_appointment",
  "description": "Books an appointment for the caller. Call it only after the caller has agreed on a time.",
  "parameters": {
    "type": "object",
    "properties": {
      "name": {"type": "string", "description": "The caller's name"},
      "time": {"type": "string", "description": "Date and time, ISO 8601"}
    },
    "required": ["name", "time"]
  }
}]}

• name: 1 to 64 characters of letters, digits, _ or - (^[a-zA-Z0-9_-]{1,64}$), unique within the session. end_conversation and transfer_to_human are reserved (6.5).
• description: at most 1,024 characters. Say when to call the tool.
• parameters: a JSON Schema for an object (type object, or left out). Supported keywords: type (object, string, number, integer, boolean, array), properties, required (names of its properties), enum (1 to 100 strings, numbers or booleans), description, items, minimum, maximum and maxLength; nested at most 6 levels deep. The keywords of everyday strict schemas, additionalProperties, title, default, examples, $schema, minLength, minItems and maxItems, are accepted and ignored: they are removed before the schema is used (GET /capabilities lists them as ignored_keywords), so validate your tool’s input yourself. Anything else (oneOf, anyOf, allOf, $ref, pattern, format …) is refused with agent.invalid_request.
• At most 32 tools, 16,000 characters for all tools together, and at most 24,000 with the instructions.

When the agent needs one, it sends tool.call. arguments is always a JSON object, and reply_id is the reply it belongs to:

{"type":"tool.call","call_id":"tc_…","name":"book_appointment",
 "arguments":{"name":"Meier","time":"2026-11-03T10:00"},"reply_id":"rp_…"}

Run the function in your backend and answer with tool.result:

{"type":"tool.result","call_id":"tc_…","output":"{\"booked\":true}"}

• Answer every call_id once. Several calls can be open at the same time.
• output is a string or any JSON value (sent as JSON, it is converted to text); at most 16,000 characters. The agent then continues and speaks the outcome.
• You have 30 seconds. After that the agent tells the caller that it did not work and the call is closed; a tool.result that arrives late is answered with agent.invalid_request (detail: “call expired”) and ignored. A call is never retried automatically. The same applies to a second result for a call_id, or for one the server never sent.
• A call stays valid when the caller interrupts the agent; a call of a reply that was interrupted or cut before it reached you is never sent.

Keep tools fast; session.status shows open_tool_calls.

6.5 Steering the conversation

Send these after session.ready; before it, the server answers with agent.invalid_request. context.add is the way to tell the agent what happened outside the call: “the caller’s account is overdue”, “the order has just shipped”. reply and say start outbound speech, for example a prompt after a long silence.

Ending a conversation. With end_conversation true (the default) the agent can end the call itself, but only on the caller’s own words: a goodbye, an explicit wish to end, or a clear “no” to the agent’s closing question; otherwise it asks first. Then you get session.end_requested with a reason (goodbye, user_request, closing_confirmed or idle), the goodbye is spoken, and the session closes with session.complete and 1000, at the latest 15 seconds later. session.complete repeats the reason as end_reason. Set end_conversation to false if your own system decides when a call ends.

Handing over to a person. Give transfer_targets (name and a short description of when to use each) and the agent can ask for a handover: you get transfer.requested with the target name and a short reason, the agent says a short handover line, and the session goes on. Your telephony makes the transfer and then ends the session with session.end.

Silence (auto mode). The clock starts with the first exchange and runs only while nobody speaks, nothing is played and no tool call is open. After idle_prompt_s the agent asks once, in the caller’s language, whether they are still there; after idle_end_s it says goodbye and the session ends (session.end_requested, reason idle).

Several languages. With several languages, recognition listens for all of them, user.transcript.final carries the language heard, and the agent answers in the caller’s language, one language per reply. For Swiss German it keeps the chosen dialect.

6.6 Limits, warnings and errors

A frame over the size limit gets agent.invalid_request (text) or agent.invalid_audio (binary) up to 65 KB and is not read; above that the socket closes with 1009, the standard WebSocket “message too big”.

Messages that break a limit are answered with an error and dropped; the session goes on. Too many frames, too many control messages, or too many reply, say and text.input, give agent.rate_limited; the rest give agent.invalid_request. Errors that do not end the session come at most once per code per second; further ones of the same code in that second are dropped silently. More than 50 refused frames within 10 seconds (over the frame rate, too large or unreadable) end the session with agent.rate_limited and the close code 1008. A refused session.start also ends the connection.

Warnings. A session.warning carries a code, and the session goes on:

Wrap the conversation up when you get the first three, or open a new session afterwards.

session.status comes every 60 seconds and when recognition changes: state (listening, thinking or speaking), recognition (ok or degraded: the agent then answers from the audio alone, and user.transcript.* stops), connected_s, remaining_s and open_tool_calls.

Errors arrive as an error message:

{"type":"error","code":"agent.invalid_request","message":"…","detail":"…","request_id":"req_…"}

Close codes. An error that ends the session comes first, then:

Log the request_id of every error and every session.complete: it is what to quote in a support request (Errors).

6.7 Billing and sandbox

A session is billed per second of connected time, from session.ready until the session ends, with no minimum, at CHF 0.33 per minute (CHF 19.80 per hour), excluding VAT. A start that never reaches session.ready costs nothing. session.complete reports duration_s (connected seconds) and billable_seconds, and, with transcripts on, the whole conversation:

{"type":"session.complete","request_id":"req_…","duration_s":184.2,"billable_seconds":184.2,
 "end_reason":"goodbye",
 "transcript":[{"role":"agent","text":"Grüezi, Garage Muster …","at_ms":420},
               {"role":"user","text":"Ich möchte einen Termin.","at_ms":3100}]}

transcript holds every user turn and what the agent actually played (an interrupted reply up to where it was heard), with at_ms since session.ready; at most 1,000 entries and 200,000 characters, then transcript_truncated is true. end_reason is present after session.end_requested, is restart when the service restarts (the session is ended properly, close 1001), and unavailable when the service cannot continue a running session (after agent.unavailable, close 1011; billed up to that moment). If your connection drops, the session ends and is billed up to that moment, but no session.complete can reach you.

Voice agents draw on the same prepaid credit as the other services (Limits and metering). Each session holds the credit for its length from the start, so sessions that run at the same time share the balance and can never overdraw it. A session needs at least one minute of credit left to start; with less, it is refused with agent.entitlement_exhausted (close 4003). If the remaining credit is shorter than your max_duration_s, the session is shortened to what the credit covers (limited_by: "credit" in session.ready). If the balance of a prepaid account cannot be read or held at the start, the session is refused with agent.unavailable (close 1011); retry shortly. A session is billed at the price in force when it started.

Connected time is billed whether or not anyone is speaking, so close the socket (session.end) when the call is over.

Sandbox. A key that starts with sv_test_ runs a synthetic session: the same events in the same order, with a synthetic transcript and a short test tone as the answer, and "sandbox": true in every event. It is never billed. Build and test your client with it; judge the voice and the dialect with a live key.

6.8 Example: browser

A complete minimal page: it asks your server for a token (6.1), captures the microphone at 16 kHz (pcm16), plays the agent’s 24 kHz pcm16 binary frames, drops audio on audio.clear and answers a tool call. The page never holds an API key; the server route /agent-token is the one shown in 6.1, whose locked preset holds the agent’s language, voice, instructions, greeting and the get_time tool.

<button id="start">Start</button> <button id="stop">Stop</button>
<pre id="log"></pre>
<script type="module">
const ENDPOINT = 'wss://api.suisse-speech.ch/v1/agent/stream';
const log = (who, text) => { document.querySelector('#log').textContent += `${who}: ${text}\n`; };

let ws, mic, micCtx, out, playAt = 0;
const playing = new Set();

// Playback: binary frames are raw 24 kHz pcm16, queued back to back.
function play(buffer) {
  const pcm = new Int16Array(buffer);
  const audio = out.createBuffer(1, pcm.length, 24000);
  const ch = audio.getChannelData(0);
  for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;
  const src = out.createBufferSource();
  src.buffer = audio;
  src.connect(out.destination);
  playAt = Math.max(playAt, out.currentTime + 0.05);
  src.start(playAt);
  playAt += audio.duration;
  playing.add(src);
  src.onended = () => playing.delete(src);
}

// audio.clear: the caller interrupted, drop everything not yet played.
function clearPlayback() {
  for (const s of playing) { try { s.stop(); } catch {} }
  playing.clear();
  playAt = 0;
}

// Capture: 16 kHz mono, 40 ms (640 samples) per binary frame.
const WORKLET = `
class Capture extends AudioWorkletProcessor {
  constructor() { super(); this.buf = new Int16Array(640); this.n = 0; }
  process(inputs) {
    const ch = inputs[0][0];
    if (!ch) return true;
    for (let i = 0; i < ch.length; i++) {
      this.buf[this.n++] = Math.max(-1, Math.min(1, ch[i])) * 32767;
      if (this.n === 640) { this.port.postMessage(this.buf.slice().buffer); this.n = 0; }
    }
    return true;
  }
}
registerProcessor('capture', Capture);`;

async function startCapture(send) {
  micCtx = new AudioContext({ sampleRate: 16000 });
  const source = micCtx.createMediaStreamSource(mic);
  const mute = micCtx.createGain();             // keeps the graph running, plays nothing
  mute.gain.value = 0;
  mute.connect(micCtx.destination);
  if (micCtx.audioWorklet) {
    const url = URL.createObjectURL(new Blob([WORKLET], { type: 'text/javascript' }));
    await micCtx.audioWorklet.addModule(url);
    const node = new AudioWorkletNode(micCtx, 'capture');
    node.port.onmessage = (e) => send(e.data);
    source.connect(node).connect(mute);
  } else {                                      // fallback for old browsers
    const node = micCtx.createScriptProcessor(1024, 1, 1);
    node.onaudioprocess = (e) => {
      const f = e.inputBuffer.getChannelData(0);
      const pcm = new Int16Array(f.length);
      for (let i = 0; i < f.length; i++) pcm[i] = Math.max(-1, Math.min(1, f[i])) * 32767;
      send(pcm.buffer);
    };
    source.connect(node).connect(mute);
  }
}

async function start() {
  out = new AudioContext({ sampleRate: 24000 });
  // Ask for the microphone first, with echo cancellation, so the agent never hears itself.
  mic = await navigator.mediaDevices.getUserMedia({
    audio: { echoCancellation: true, noiseSuppression: true, channelCount: 1 },
  });
  // A token from your server (6.1): valid for 60 s and one session, never the API key.
  const { token } = await (await fetch('/agent-token', { method: 'POST' })).json();
  ws = new WebSocket(ENDPOINT, ['suisse.v1', 'key.' + token]);
  ws.binaryType = 'arraybuffer';
  // Language, voice, instructions, greeting and the get_time tool come from the token's
  // locked preset (6.1); the page sends only the audio formats.
  ws.onopen = () => ws.send(JSON.stringify({
    type: 'session.start',
    input:  { encoding: 'pcm16', sample_rate_hz: 16000 },
    output: { encoding: 'pcm16', sample_rate_hz: 24000, transport: 'binary' },
  }));
  ws.onmessage = async (e) => {
    if (e.data instanceof ArrayBuffer) return play(e.data);      // the agent's voice
    const m = JSON.parse(e.data);
    switch (m.type) {
      case 'session.ready':
        await startCapture((buf) => ws.readyState === WebSocket.OPEN && ws.send(buf));
        break;
      case 'audio.clear': clearPlayback(); break;
      case 'user.transcript.final': log('You', m.text); break;
      case 'reply.done': log('Agent', m.transcript); break;
      case 'tool.call': {
        const result = { time: new Date().toLocaleTimeString('de-CH') };
        ws.send(JSON.stringify({ type: 'tool.result', call_id: m.call_id, output: JSON.stringify(result) }));
        break;
      }
      case 'error': log('Error', `${m.code} ${m.message}`); break;
      case 'session.complete': log('Done', `${m.billable_seconds} s billed`); break;
    }
  };
  ws.onclose = stop;
}

function stop() {
  if (ws?.readyState === WebSocket.OPEN) ws.send(JSON.stringify({ type: 'session.end' }));
  mic?.getTracks().forEach((t) => t.stop());
  micCtx?.close(); out?.close();
  mic = micCtx = out = undefined;
}

document.querySelector('#start').onclick = start;
document.querySelector('#stop').onclick = stop;
</script>

6.9 Example: Node.js and a telephone line

A telephone bot that passes G.711 µ-law at 8 kHz straight through. call stands for your telephony connection (a media gateway or a SIP stack): it delivers the caller’s audio and plays audio back to the caller.

import WebSocket from 'ws';

const ws = new WebSocket('wss://api.suisse-speech.ch/v1/agent/stream', {
  headers: { 'X-API-Key': process.env.SUISSE_SPEECH_KEY },
});
let ready = false;

ws.on('open', () => ws.send(JSON.stringify({
  type: 'session.start',
  language: 'de-CH', dialect: 'bern', voice: 'anna',
  instructions: 'Du bist die Terminvergabe der Praxis Muster. ' +
                'Sprich kurz und freundlich, frage nach dem Namen und biete freie Termine an.',
  greeting: 'Grüezi, Praxis Muster. Wie kann ich helfen?',
  vocabulary: ['Muster', 'Dr. Brunner'],
  input:  { encoding: 'mulaw', sample_rate_hz: 8000 },
  output: { encoding: 'mulaw', sample_rate_hz: 8000, transport: 'binary' },
  turn_taking: { mode: 'auto', interruptions: true, patience: 'normal' },
  max_duration_s: 600,
  tools: [{
    name: 'free_slots',
    description: 'Lists the free appointment slots of the next days.',
    parameters: { type: 'object', properties: {} },
  }, {
    name: 'book_appointment',
    description: 'Books an appointment. Call it only after the caller has agreed on a time.',
    parameters: {
      type: 'object',
      properties: { name: { type: 'string' }, time: { type: 'string', description: 'ISO 8601' } },
      required: ['name', 'time'],
    },
  }],
})));

// The caller's audio: Buffers of µ-law, in real time.
call.on('audio', (frame) => { if (ready && ws.readyState === WebSocket.OPEN) ws.send(frame); });
call.on('hangup', () => ws.readyState === WebSocket.OPEN && ws.send(JSON.stringify({ type: 'session.end' })));

const tools = {
  free_slots: async () => calendar.freeSlots(),
  book_appointment: async ({ name, time }) => calendar.book(name, time),
};

ws.on('message', async (data, isBinary) => {
  if (isBinary) return call.play(data);                  // the agent's voice, 40 ms per frame
  const m = JSON.parse(data.toString());
  switch (m.type) {
    case 'session.ready': ready = true; break;
    case 'audio.clear': call.clearPlayback(); break;     // the caller interrupted
    case 'tool.call': {
      let output;
      try { output = JSON.stringify(await tools[m.name](m.arguments)); }
      catch (err) { output = JSON.stringify({ error: 'not available' }); }
      ws.send(JSON.stringify({ type: 'tool.result', call_id: m.call_id, output }));
      break;
    }
    case 'session.warning': console.warn(`ends in ${m.seconds_left} s`); break;
    case 'error': console.error(m.code, m.message, m.request_id); break;
    case 'session.complete': console.log(`billed ${m.billable_seconds} s (${m.request_id})`); break;
  }
});

ws.on('close', (code) => { ready = false; call.hangup?.(); });