Skip to content

PMAT-3797: CRUX serve verb — apr serve, llama-server and ollama through ONE OpenAI client, streaming + non-streaming - #3802

Closed
noahgift wants to merge 1 commit into
crux-3739-chatfrom
crux-3739-serve
Closed

noahgift wants to merge 1 commit into
crux-3739-chatfrom
crux-3739-serve

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

CRUX slice 4: the serve verb

Closes #3797. Stacked on #3796 (→ #3789 → #3767), so this diff is crux-3739-chat...HEAD.

keep-open: #3739 stays open. TTFT and decode rate via apr test llm bench, max context, code, thinking ON, consumer rungs and T-1 wiring are later increments; all are named in every receipt's not_covered.

What it adds

Measured on lambda

cpu lane, qwen2.5-coder-1.5b Q4_K_M, --verbs run,serve: PASS, 9 cells, 9 GREEN. apr serve, llama-server@d1d3c3396 and ollama 0.33.2 each answered every golden prompt in both modes, and nothing was left running.

  • The first run found a teardown bug. Every request was answered, but the cell hung: a bare wait after killing apr serve blocked on llama-server. Teardown now kills every server first and waits on those PIDs only. The rerun finished unaided.
  • apr serve renders a different prompt. It answers 2+2 from an 18-token prompt with "4", where llama-server and ollama send 36 tokens (the GGUF template's default system prompt) and answer "2 + 2 equals 4.". This is recorded, not judged here; the byte-equal tmpl rows against HF will judge it.

Verified

  • bash scripts/check_crux_inference_judge.sh: 44 ok, 0 broke. Serve rows: stream and non-stream are distinct GREEN cells; apr serve answering nothing (connection refused) while llama-server answers is RED. Two serve mutants each break a row.
  • bash scripts/guard_tree.sh --no-cargo: 77 checks, 0 failed on fa8623a5d. check_no_competing_harnesses.sh: count=0. The verb name serve run and my first rate-field names had tripped its text test, so the fields are now decode_rate with a separate rate_unit.

🤖 Generated with Claude Code

…gh ONE OpenAI client, non-streaming and streaming; the key's additive mode; modes declared on the serve run correspondence entry

Refs #3797

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Contributor Author

Every commit is already in release/0.69.1-batch-2 (git cherry: 0 unfolded). It reaches main through the release batch PR #3835. Closing as folded, not rejected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant