Conversation
…gh ONE OpenAI client, non-streaming and streaming; the key's additive mode; modes declared on the serve run correspondence entry Refs #3797 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Every commit is already in release/0.69.1-batch-2 (git cherry: 0 unfolded). It reaches main through the release batch PR #3835. Closing as folded, not rejected. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CRUX slice 4: the
serveverbCloses #3797. Stacked on #3796 (→ #3789 → #3767), so this diff is
crux-3739-chat...HEAD.keep-open: #3739 stays open. TTFT and decode rate via
apr test llm bench, max context,code, thinking ON, consumer rungs and T-1 wiring are later increments; all are named in every receipt'snot_covered.What it adds
--verbs run,serveruns one serve cell per model under one hold of the GPU lock.apr serve runandllama-servereach load the model once, next to the running ollama daemon.scripts/lib/crux_openai_client.py, non-streaming and streaming.crux_openai_client.pyis the one OpenAI client: the same body for every server, and the row-contract JSON out. It reads no clock and computes no rate; TTFT and throughput areapr test llm bench's job (PERF-009).mode(nonstream | stream), and apr serve's cells carrybackend_verified: false.serve runentry declaresmodes: [nonstream, stream], with per-engine modes, so pv never names a verb.Measured on lambda
cpu lane, qwen2.5-coder-1.5b Q4_K_M,
--verbs run,serve: PASS, 9 cells, 9 GREEN. apr serve, llama-server@d1d3c3396 and ollama 0.33.2 each answered every golden prompt in both modes, and nothing was left running.waitafter killing apr serve blocked on llama-server. Teardown now kills every server first and waits on those PIDs only. The rerun finished unaided.tmplrows against HF will judge it.Verified
bash scripts/check_crux_inference_judge.sh: 44 ok, 0 broke. Serve rows: stream and non-stream are distinct GREEN cells; apr serve answering nothing (connection refused) while llama-server answers is RED. Two serve mutants each break a row.bash scripts/guard_tree.sh --no-cargo: 77 checks, 0 failed onfa8623a5d.check_no_competing_harnesses.sh: count=0. The verb nameserve runand my first rate-field names had tripped its text test, so the fields are nowdecode_ratewith a separaterate_unit.🤖 Generated with Claude Code