PMAT-3795: CRUX chat verb — a two-turn conversation through apr, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn - #3796
Closed
noahgift wants to merge 1 commit into
Conversation
…at via stdin, llama.cpp and ollama via a pty driver (neither takes turns from a pipe), judged on the final turn Refs #3795 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Every commit is already in release/0.69.1-batch-2 (git cherry: 0 unfolded). It reaches main through the release batch PR #3835. Closing as folded, not rejected. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CRUX slice 3: the
chatverbCloses #3795. Stacked on #3789 (→ #3767), so this diff is
crux-3774-verb-correspondence...HEAD.keep-open: #3739 stays open; serve, code, thinking ON, consumer rungs and T-1 wiring are later slices. #3794 is the apr chat backend defect this slice found; it's filed, not fixed here.
What it adds
--verbs run,chatruns a two-turn conversation per model: "What is 2+2?" then "Now multiply that by 3.". It's judged on the final reply (12, which needs the first turn's4carried over), and every reply is recorded.chatwithoutrundeclines, because the positive control lives inrun.apr chat, one user turn per stdin line.scripts/lib/crux_pty_chat.py, a stdlib pseudo-terminal driver. It waits for each CLI's prompt marker, sends one turn, captures the reply, and writes the row-contract JSON the judge already reads.gen --verb chat(infra-3c's engines, not merged yet).Assistant:transcript. apr's chat cells carrybackend_verified: false, becauseapr chatreports no backend (apr chat --no-gpu still uploads the GGUF to CUDA (VRAM held outside the GPU lock), and apr chat reports no machine-readable backend #3794);not_coverednames it. Run-verb token parity isn't claimed for chat.Why a pty
Measured on lambda:
llama-cliloops on empty>prompts until killed;ollama runreads the whole pipe as one prompt and answers both turns in one reply.Neither takes turns from a pipe, so a piped "multi-turn" test of either would have been a single-turn test wearing a label.
Found while building it: #3794
apr chat --no-gpustill uploads the GGUF to CUDA: it holds VRAM outside the GPU lock, while generation does honour--no-gpu. Andapr chatreports no machine-readable backend.Verified
bash scripts/check_crux_inference_judge.sh: 42 ok, 0 broke. The chat rows are GREEN, RED (apr "9" vs llama.cpp "12"), RED (no Assistant turn), and the turn/backend/parity record. Three chat mutants each break a row.--verbs run,chat): PASS, 4 GREEN. In the chat cell apr, llama.cpp@d1d3c3396 and ollama 0.33.2 each gave["2 + 2 equals 4.", "4 * 3 equals 12."], and no ollama import was left behind.bash scripts/guard_tree.sh --no-cargo: 77 checks, 0 failed on05685a2de.check_no_competing_harnesses.sh: count=0.🤖 Generated with Claude Code