Serve a browser test client at /dev - #1
Merged
Conversation
Every latency figure sonari had was a single observation, and there was no accuracy figure at all: the existing harness drove one recording and printed one line. The optimisation log says as much — "percentiles wait for a golden set". This adds the golden set and the instrument that reads it. **The harness.** `sonari-eval generate` synthesises fifteen clips and writes a manifest; `sonari-eval run` plays them through the system and writes a report of word error rate, the eight markers of ADR-0010 as percentiles, and how many samples failed. Two solvers sit behind one trait: one drives the components in process and runs anywhere with an API key; one drives a running service as a LiveKit caller and reads the markers back from the call's own events. The report names which produced it, because only one of them exercises endpointing. **The markers.** ADR-0010 asked for eight per turn carried as elapsed values. Forty-odd lifecycle events existed and none carried time. The runtime now emits one `speech_turn_latency` event per turn with every marker and the two derived figures, computed where the timings are known rather than by each reader. Two of them are measured inside the model's streaming loop, which is the only place they exist. `audio_first_frame` remains unwired; the report says so. **The clips are synthesised, not recorded,** because the set measures timing and a person cannot produce a 400 ms pause reliably. The audio is committed rather than regenerated: the assembly is deterministic but the synthesiser is not, so regenerating later would silently produce a different set. The harness records what happened and asserts nothing about what should have. That principle earned itself during the work: the clip set was built on the premise that a long pause gets committed early, and the premise looks false — see docs/tickets/0001. A harness that checked expectations would have reported failing clips instead of a finding about the system. Two runs against the real providers corrected the harness itself. Recognition applies inverse text normalisation, returning `8291` and `7:30` where the reference says "eight two nine one" and "seven thirty", so the first run reported 13.7% error for recognition that was perfect; the normaliser now folds numbers and colloquial contractions, and the set scores zero. CI gains `cargo test -p harness`. Tests needing credentials or a running stack skip themselves, so CI makes no network calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
The harness waited for the bot's audio track before sending the caller's audio. The runtime does the opposite: it subscribes to the caller first and publishes its own track only afterwards. Each side waited for the other, which looked like the bot intermittently missing the caller's track. The gate now waits for the bot to be present in the room, which is the actual precondition and one a client can observe. With that out of the way the real behaviour is visible, and ticket 0002 is rewritten to describe it: recognition closes about a second into the call, every frame after that is dropped, no speech is ever detected, and the next call in the batch gets no runtime at all. The media path was never the problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
The recognition WebSocket never opens: the log that marks a completed handshake never appears, and neither does any failure from the socket task. A cancelled task logs nothing, and the only thing cancelling these is the session-close path — so the session is being closed about a second in, taking a half-open socket with it, and the frames that had queued behind it all fail at once. The network is not involved; ElevenLabs answers in 150 ms from the same compose network. The open question is now what closes the session, not why recognition disconnects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
The recognition task logged when it started and when it failed, but nothing when it returned cleanly and nothing while it was connecting. A task cancelled mid-handshake therefore left no trace at all, which is exactly what has been happening: the socket never opens, the task is cancelled before any timeout fires, and the only symptom is every frame afterwards being dropped. Three lines close that gap — the open attempt, the elapsed time when it succeeds, and a clean exit — along with a warning when a speech session fails, which until now was recorded only as a call event. With them the failure is narrowed to the service image: the same binary, key and network transcribe correctly from the dev container, and the service container reaches api.elevenlabs.io over TLS, but inside it the WebSocket handshake never completes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
Two rustls crypto backends reach this binary — different dependencies pull different ones — so rustls refused to guess and panicked on the first TLS handshake. That handshake was the recognition WebSocket's, and the panic happened inside a spawned task, so it went to stderr and never through tracing. Every structured log showed the same thing: recognition simply ceased to exist, frames were dropped, the session failed, and the next call got no runtime. Installing the provider once at startup is what the panic message asks for, and is sturdier than hoping feature unification stays lucky across a workspace this size. After it, a live call opens recognition in 146 ms, detects speech, flushes an utterance and returns a transcript. That also retires the reasoning in ticket 0001, which was formed on a system where recognition had never worked: the flush branch is plainly reachable. Two behaviours remain open and are recorded in ticket 0002: a session that closes while its turn is still in flight, and a call that leaves the next one without a runtime. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
Two failures that end a call were reported at debug: a pipeline task that returned an error, and one that finished without being asked to. Both are now warnings — the same lesson as the rustls panic, which was invisible for the same reason. The inbound audio drain also says why it stopped, since "the caller's audio stopped arriving" has several causes and they lead different places. Those lines localise the remaining bug without solving it. With recognition working, a call now reaches a transcript and is then torn down two seconds in, while the caller is still speaking: the pipeline's shutdown senders are all dropped, which is neither the stop path nor the reap path, and neither failure warning fires. Ticket 0003 records the timings and where to look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
The media plane ran as a spawned task whose result nobody read, and every
control-plane error inside it propagated with `?` out of the worker loop. A
single stale fact —
publish runtime event failed (NotFound): speech session not found
— therefore ended the task, dropping the worker, its map of active runtimes, and
every shutdown sender with them. The call in progress was torn down mid-turn and
no later call was ever claimed, while the process went on answering /healthz as
if nothing had happened.
The plane is now supervised and says what became of it. A rejection about one
session is recorded and skipped rather than ending the loop, and a retryable
failure leaves the action queued instead of propagating. One call's problem stays
one call's problem.
Two consecutive probe calls now complete with a spoken reply, and a fifteen-clip
live evaluation runs every clip.
That is enough for the evaluation set to do its job, and its first live run
confirms ticket 0001: speech is detected, and the utterance is never flushed
until the caller hangs up — twenty seconds after a 4.2 s clip, against a 700 ms
setting. The flush that does appear comes from the close path, not from silence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
It was hardcoded to zero, and the comparison behind it is a mean of absolute sample values, which is never below zero. Every frame therefore counted as voice, silence was never observed, and a turn ended only when the caller hung up — twenty seconds after a four-second clip, against a 700 ms setting. It is now a setting, defaulting to 300: above a −60 dBFS noise floor, which lands near 33 on this scale, and well below the thousands that speech reaches in the calls measured so far. It is a starting point for the evaluation set to tune, not a tuned value. Four tests pin the behaviour, including the one that would have caught this when it was written. Live verification is still pending — the Docker daemon went down before a call could confirm that a turn now ends on silence rather than on hang-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
The threshold change could not be judged: with it set to 300 no speech is detected on a live call at all. Logging the loudest frame the comparison saw each second says why — the peak reaching it is 9, on a clip whose frames peak at 16823 and hold near 9000 through the speech. The audio is not quiet, it is absent. It is absent only sometimes; the same clip has produced a real transcript on an earlier call. So delivery is intermittent, which is ticket 0004, and it blocks judging ticket 0001's threshold. The peak is logged per second rather than a sampled frame: speech is a minority of frames, and a random sample reports silence during speech often enough to send an afternoon in the wrong direction — which it did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
The caller was talking over the hello. The service drops inbound frames while its
own turn is pending, and both eval clients started speaking the moment they saw
the bot's track — so whether a clip was heard depended on whether it happened to
land in the barge-in path. That is what looked like audio vanishing in transport.
The live solver now waits until the bot's audio has been quiet for 700 ms before
speaking, which is what a caller does and keeps this measuring an ordinary turn
rather than a barge-in.
With that, the first complete live evaluation:
hangover cost p50 700 ms exactly silence_flush_ms
system response p50 854 p95 976
perceived p50 1553 p95 1676
pause-400 1 utterance wer 0.00
pause-600 1 utterance wer 0.00
pause-800 1 utterance wer 0.20 endpoint 185 ms into the gap
pause-1200 2 utterances wer 0.22
Turns end on silence after the configured wait rather than at hang-up, and the
boundary sits where the setting says it does: pauses under 700 ms survive, longer
ones lose words or split the utterance. Silence and the cough still open no turn.
That closes ticket 0001, whose answer this set was built to produce, and ticket
0004. The report is kept as the first live baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
It reported `perceived_response_ms: 0.0` on every run, because a subscribed track delivers frames whether or not anyone is speaking and the probe's ear was a byte counter. Silence was an answer, and the answer arrived instantly. Frames now carry their loudness as well as their size. Only audible audio counts as a reply, and the probe waits before speaking rather than talking over the greeting — which the service drops on the floor, so its utterance was often never heard at all. The measurement is now real: 280 ms to a 1.3 s reply, against a constant zero before. The waiting is not yet detection, in either client. Frames keep arriving during silence, so "wait until they stop" waits the whole window, and the probe finds the agent's track carries zeros throughout the greeting and real audio only afterwards. Both are therefore waiting on a clock, which works and is slow. Recorded in ticket 0004 rather than left implied by a comment that claims more than the code does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
The probe now records the loudest frame on the agent's track each second, which is what separates "the call is quiet" from "the client is not listening yet". It shows the greeting arriving as zeros while the service logs it as synthesised and played, and the reply arriving loud a moment later. Either the greeting is never written to the room — in which case a caller hears silence until they speak — or the client's audio stream is not delivering for its first few seconds, in which case every client here under-reports the start of a call. Ticket 0005 says how to tell the two apart; guessing between them would have been the third time this session that a plausible story turned out to be the wrong one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
A second run heard it plainly — peaks of 12356 and 10522 in the seconds the service says it was speaking. In the run where it was missed, the client's audio stream had produced no frames at all for its first three seconds, and the greeting came and went inside that window. So nothing is wrong with the greeting, and the intermittency was never in the audio: sometimes the stream is delivering by the time the agent speaks, sometimes it is not. Both eval clients wait before speaking, which stays right, but neither can rely on hearing the greeting end. Left in the ticket, unmeasured: the caller sent nothing but noise floor from second five to twenty-four and the agent never spoke, though `silence_force_agent_ms` is eight seconds. An evaluation set that only ever speaks cannot see that timer; it would want a clip of its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
A set that only ever speaks cannot see what happens when nobody does. `idle-force-agent` is twelve seconds of noise floor, and the report counts the turns the agent started on its own — the greeting is one; a second would mean it filled the silence. The first run said the agent greeted and never spoke again, which looked like `silence_force_agent_ms` failing to fire after eight seconds. Reading the code before writing that up: the setting is not what its name says. Its deadline is set at commit time as `silence_force_agent_ms - silence_flush_ms` and is spent waiting for recognition's final result before going ahead with the partial. A silent caller gets no turn from it, and the agent was behaving correctly. So the config comment now describes the code, the report states the count without implying a verdict, and the clip stays — the behaviour is worth watching whether or not a setting promises it. This is the fifth time this session a plausible reading of a symptom was wrong, and the first time the harness's own rule — record what happened, assert nothing — was what stopped it becoming a bug report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqX1vTHXSSaHi2ffKG1EoE
The harness reads the product design document by a fixed name, docs/product.md, at the start of every task. Renaming it removes the per-task lookup rather than teaching each reader a project-specific name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016n12Y7LX1cWJHFRV9Jsvss
There was no way to talk to a running deployment. POST /api/session returns a
token and POST /api/call/{character_id}/start returns everything needed to join
a LiveKit room, but nothing in the repository carried either into a browser, and
the eval harness deliberately never opens one. Trying a change by hand meant
writing a throwaway page each time.
Three pieces:
- GET /api/personas publishes the persona list. A call is started by an id
derived from the persona's name, and nothing published it, so a person had to
read an id out of the logs to call anything. It is read through a new port on
character-context, implemented by the same ConfigPersonas the call path
resolves ids through, so a listed id is an id a call accepts. Unauthenticated,
beside /api/session: the token is free to mint and identifies nobody.
- GET /dev serves a single page that walks the ordinary contract — session, list
personas, start call, join the room, end call. Same-origin with the API, so no
CORS layer exists anywhere.
- The page and a vendored copy of the LiveKit browser SDK, pinned at 2.21.0, are
compiled into the binary. Compose mounts only models/ and sonari.toml; adding
a third mount for a test page would make the test tool a deployment concern.
Neither the Dockerfile nor docker-compose.yml changes.
product.md said a web client was not built. It now says a web product client is
not built, and names this one a test tool. Android remains the product surface.
Decisions: ADR-0018, ADR-0019, ADR-0020.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016n12Y7LX1cWJHFRV9Jsvss
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There was no way to talk to a running deployment by hand.
POST /api/sessionreturns a token andPOST /api/call/{character_id}/startreturns everything needed to join a LiveKit room, but nothing in the repository carried either into a browser, and the eval harness deliberately never opens one.What this adds
GET /api/personas— a call is started by an id derived from the persona's name (id_for), and nothing published it, so starting a call by hand meant reading an id out of the logs. Read through a newCharacterCatalogReadPortoncharacter-context, implemented by the sameConfigPersonasthe call path resolves ids through, so a listed id is an id a call accepts. Unauthenticated, beside/api/session. Three fields on the wire — prompts and the synthesis voice stay off it.GET /dev— one page, no framework, no build step. Enter auid, pick a persona, call, talk, hang up. It walks the ordinary contract and gets no privilege. Same-origin with the API, so no CORS layer exists anywhere. Every failure is written into the page as text.Vendored LiveKit browser SDK, pinned at 2.21.0 (Apache-2.0, licence and provenance in
crates/api/assets/dev-client/README.md). The page and the SDK are compiled into the binary withinclude_str!: compose mounts onlymodels/andsonari.toml, and a third mount for a test page would make the test tool a deployment concern.Dockerfileanddocker-compose.ymlare unchanged.Documents
product.mdsaid a web client was not built. It now says a web product client is not built, and names this one a test tool — Android remains the product surface.architecture.mdgains the route and the two unauthenticated endpoints.Decisions: ADR-0018 (the client lives in the binary), ADR-0019 (vendor the SDK), ADR-0020 (publish the persona list).
Tests
16 new cases, all Rust — the page is for a person, and no browser-side test tooling is introduced. Two of them run against
build_router_with_modulesrather than a router built in the test, because every other test would pass if the application never merged these routes.Note: this branch also carries
4f41b77, renamingdocs/prd.mdtodocs/product.md.🤖 Generated with Claude Code
https://claude.ai/code/session_016n12Y7LX1cWJHFRV9Jsvss