Building a Swiss German Voice Bot: The Latency Budget

21 September 2026 · 8 min read

A practical target for a Swiss German voice bot is that the caller hears the first word of the reply within about 1.5 seconds of finishing their sentence, or about 2 seconds when the reply is spoken in dialect. That time splits into five stages: turn detection, the final transcript, your language model, the first audio of the reply and the phone line. Speech recognition and synthesis take about 0.6 seconds of it, up to about 1.2 seconds with a dialect reply; your turn detection and your LLM decide the rest.

Where the time goes

In conversation between people, the gap between turns is usually only a few hundred milliseconds, so a bot’s pauses stand out quickly. Between the caller’s last word and the bot’s first, five things happen in sequence:

1. Turn detection. Your application decides that the caller has finished, typically after a short silence, and ends the audio for this turn.
2. Final transcript. Speech-to-text delivers the final text of the turn.
3. Your LLM. Your language model reads the transcript and starts writing the reply.
4. First audio. Text-to-speech turns the first sentence into sound.
5. The line. The audio travels through your telephony stack and a playout buffer to the caller.

A latency budget with measured figures

The speech figures come from our production service, measured on a warm service on 3 September 2026:

When the reply is rendered into Swiss German dialect, the first audio arrives in under a second. The other rows of the budget belong to you: they depend on your turn detection, your LLM and your telephony. Plan with p95 values rather than averages, because the slow turns are the ones callers notice.

As laid out, the budget misses the target. The next section closes the gap.

Getting under the target

Overlap turn detection and the LLM

Partial transcripts arrive every 250 ms and, at the median, lag the audio by less than a tenth of a second. By the time your turn detection fires, the latest partial covers almost everything the caller said. Start your LLM on it: if the final transcript matches, the LLM has been working while turn detection was still waiting; if it differs, discard the draft and start again.

Stream the reply and keep the first sentence short

Do not wait for the complete reply. Push the LLM’s output into the synthesis socket as it arrives; synthesis renders passage by passage at sentence boundaries, so the caller hears the first sentence while the model is still writing the third. Ask your LLM to open with a short sentence, such as Gern, das schaue ich nach., and send text.flush when a partial sentence should go out without waiting.

Prepare what never changes

Greetings, menu prompts and holding phrases do not need realtime synthesis. Produce them once in studio quality with a batch request and play them from your own storage. The API keeps no copies and synthesises identical texts afresh each time, so a file you store is the fastest audio there is.

Ask for the format the line uses

Request G.711 µ-law or A-law at 8 kHz directly ("format": "mulaw" or "alaw"), so there is no transcoding step, and add "transport": "binary": audio then arrives as raw binary frames, without the roughly 36 % overhead of base64 inside JSON. On the way in, realtime recognition expects 16 kHz mono PCM16, so convert the 8 kHz call audio before you send it.

Keep connections ready

Open the recognition session for the next turn and the synthesis socket a moment before you need them, so no handshake sits on the critical path. Opening realtime sessions has its own limit of 600 per minute per key, separate from ordinary requests.

Plan for concurrency

Your plan sets a ceiling on simultaneous requests. Beyond it, the service answers 429 with Retry-After instead of queueing, so the sessions that got in keep their latency. In our measurements, 25 and 30 simultaneous realtime sessions ran without refusals at a first-audio p95 of 0.78 s and 0.85 s. Read your ceiling from GET /v1/capabilities and back off properly on 429 (limits).

Swiss details that shape the design

Choose the dialect per line or region

Recognition writes Swiss German as Standard German and reports the language, not the dialect. The dialect of the reply is therefore your decision, made per deployment, phone line or region, or taken from the customer record: Bernese for a hotline in Bern, for example. List it in the recognition candidates, such as de-CH-bern,fr-CH,en: the code comes back exactly as listed and can be passed unchanged to synthesis. A caller who switches to French gets fr-CH in the same voice, because every voice speaks every language.

Read numbers the Swiss way

Phone numbers, reference numbers and amounts are details a caller must get exactly right. Mark them up: <say-as interpret-as="telephone"> reads a number in the Swiss 3-3-2-2 grouping in the chosen dialect, digits reads codes digit by digit, and currency turns CHF 1250.50 into francs and rappen. Markup works on the stream too: a text.append that holds a complete <speak> document is expanded at once.

Let callers interrupt

Keep recognition running while the bot speaks. When partials show that the caller has started talking, stop playback and close the synthesis socket. Realtime sessions are not resumable, so the next reply simply opens a new one.

A minimal pipeline

The sketch below handles one caller turn: it streams the caller’s audio into recognition and the reply into synthesis. For the next turn, open a new recognition session. yourLlm.stream() stands for your language model’s streamed output, and call for your telephony connection.

import WebSocket from 'ws';

const API = 'wss://api.suisse-speech.ch/v1';
const headers = { 'X-API-Key': process.env.SUISSE_SPEECH_KEY };

// Listen: send 16 kHz mono PCM16 as binary frames, then {"type":"audio.end"} when the turn ends.
function listen(onFinal) {
  const stt = new WebSocket(`${API}/stt/stream`, { headers });
  stt.on('open', () => stt.send(JSON.stringify({
    type: 'config',
    lang: 'de-CH-bern,fr-CH,en',          // the dialect travels with the code
    vocabulary: ['Halbtax', 'Wankdorf'],
  })));
  stt.on('message', (data) => {
    const msg = JSON.parse(data.toString());
    if (msg.type === 'transcript.final') onFinal(msg.text, msg.lang);
  });
  return stt;
}

// Speak: stream the reply into synthesis while your LLM is still writing it.
async function speak(chunks, language, play) {
  const tts = new WebSocket(`${API}/tts/stream`, { headers });
  await new Promise((resolve) => tts.once('open', resolve));
  tts.on('message', (data, isBinary) => { if (isBinary) play(data); }); // G.711 µ-law, in order
  tts.send(JSON.stringify({
    type: 'session.start', voice: 'anna', language,
    format: 'mulaw', sample_rate_hz: 8000, transport: 'binary',
  }));
  for await (const text of chunks) tts.send(JSON.stringify({ type: 'text.append', text }));
  tts.send(JSON.stringify({ type: 'session.end' }));
  return tts; // close it if the caller interrupts
}

const stt = listen((text, lang) => speak(yourLlm.stream(text), lang, (audio) => call.play(audio)));
call.onAudio((pcm16k) => { if (stt.readyState === WebSocket.OPEN) stt.send(pcm16k); });

The text-to-speech and speech-to-text documentation lists every message and field; the API reference has the full schemas.

Measure your own budget

Log a timestamp per turn at each hand-over: the caller’s last audio frame, audio.end sent, transcript.final received, first LLM token, first text.append, first audio frame received and first audio played. Chart p50 and p95 per stage. Our figures are a starting point, not a promise for your network: measure again whenever you change anything on the audio path.

What it costs

Realtime speech-to-text costs CHF 2.20 and realtime text-to-speech CHF 6.85 per hour of audio, billed per second. A five-minute call with recognition running throughout and two minutes of bot speech comes to about CHF 0.41 for speech (5 × 0.037 + 2 × 0.114), plus your LLM and telephony (pricing). Every new account gets 60 free minutes, and sandbox keys are never billed, so your automated tests cost nothing. See text-to-speech for voices and dialects.

FAQ

How fast should a voice bot answer?

Aim for the first audio within about 1.5 seconds of the caller’s last word, or about 2 seconds with a dialect reply. Measure each stage at p95, not on average.

How fast is Swiss German text-to-speech?

In realtime, the first audio of a reply typically arrives in about half a second (measured p50 0.48 s, p95 0.58 s), and in under a second when the reply is rendered into Swiss German dialect.

Can the bot answer in the caller’s dialect?

It answers in the dialect you choose. Recognition reports the language, not the dialect, so pick the dialect per line, region or customer record. Text-to-speech renders your Standard German reply into Zürich, Bern, Basel or Lucerne German, or into one of 14 further regions on a best-effort basis.

Which audio format should a phone bot use?

For the bot’s voice, G.711 µ-law or A-law at 8 kHz, whichever your line uses, with binary transport. For realtime recognition, 16 kHz mono PCM16.

Should the LLM output be streamed into speech synthesis?

Yes. Send text as it arrives. Synthesis renders sentence by sentence, so the caller hears the first sentence while the rest is still being written.

What happens when the caller interrupts?

Keep recognition running during the bot’s turn. When partials show speech, stop playback, close the synthesis session and answer the new turn.