Publish the latency the system was measured at - #3
Open
coderloganli wants to merge 1 commit into
Open
Conversation
README and architecture.md claimed sub-2s and carried no number. The figures existed — a live run over the evaluation set on 2026-08-15 — and had never left the JSON. 854 ms system response, 1553 ms perceived, at p50; 976 and 1677 at p95. Corpus WER 4.4%. Both latency figures are reported together wherever either appears (ADR-0010), and every figure names the run it came from. That run's limits are stated beside the numbers rather than left for a reader to discover: one epoch, 15 clips where the set now holds 16, and no record of whether the build was a release one. Re-running it at three epochs is a ticket of its own, for after the work still to come. check-published-figures.sh, in the CI docs job, holds the documents to the run: a figure has to be the figure the newest run measured, under the label it measured it under, and the document has to cite that run. The next run turns CI red until the three documents are rewritten, which is what makes the re-run ticket land rather than linger. BatchReport gains `build`, so the release-build rule product.md states is checkable after the fact instead of taken on trust. Reports written before it still deserialise. Android moves to what is still to be built: product.md listed it as the client and architecture.md called it the client in production, and no Android source has ever been in this repository. The v1/v2 labels go with it — this project has no version scheme. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNYsFLhu6UE4Q4GY7nG2ng
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
README and
docs/architecture.mdclaimed sub-2s and carried no number. The figures existed — the live run of 2026-08-15 over the evaluation set — and had never left the JSON.Corpus WER 4.4%, per-clip p50 0% and p90 18%.
Published in README §Measuring it,
docs/architecture.md§7, and a new newest entry incrates/harness/OPTIMISATION-LOG.md. That run's limits are stated beside the figures: one epoch, 15 clips where the set now holds 16, and no record of the build profile.edge-8khz-stereofailed by design — 8 kHz stereo against a 16 kHz mono pipeline — and the percentiles are over the other 14. Re-running at three epochs is its own ticket, for after the work still to come.scripts/check-published-figures.shjoins the CI docs job: a published figure has to be the figure the newest run measured, under the label it measured it under, and the document has to cite that run. Committing the next run turns CI red until all three documents are rewritten.BatchReportgainsbuild, so the release-build ruleproduct.mdstates is checkable after the fact. Reports written before it still deserialise.Android moves to what is still to be built —
product.mdlisted it as the client andarchitecture.mdcalled it the client in production, and no Android source has ever been in this repository. The v1/v2 labels go with it.Not run locally: the full workspace suite needs the dev container and Docker was not up.
cargo test -p harness(70 passed), clippy and fmt were run natively, and the change touches no crate outsideharness.🤖 Generated with Claude Code
https://claude.ai/code/session_01HNYsFLhu6UE4Q4GY7nG2ng