Skip to content

PMAT-3795: CRUX chat verb — a two-turn conversation through apr, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn - #3796

Closed
noahgift wants to merge 1 commit into
crux-3774-verb-correspondencefrom
crux-3739-chat
Closed

noahgift wants to merge 1 commit into
crux-3774-verb-correspondencefrom
crux-3739-chat

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

CRUX slice 3: the chat verb

Closes #3795. Stacked on #3789 (→ #3767), so this diff is crux-3774-verb-correspondence...HEAD.

keep-open: #3739 stays open; serve, code, thinking ON, consumer rungs and T-1 wiring are later slices. #3794 is the apr chat backend defect this slice found; it's filed, not fixed here.

What it adds

  • --verbs run,chat runs a two-turn conversation per model: "What is 2+2?" then "Now multiply that by 3.". It's judged on the final reply (12, which needs the first turn's 4 carried over), and every reply is recorded. chat without run declines, because the positive control lives in run.
  • How each engine is driven:
    • apr: apr chat, one user turn per stdin line.
    • llama.cpp and ollama: scripts/lib/crux_pty_chat.py, a stdlib pseudo-terminal driver. It waits for each CLI's prompt marker, sends one turn, captures the reply, and writes the row-contract JSON the judge already reads.
    • hf and llamafile: gen --verb chat (infra-3c's engines, not merged yet).
  • Judge: it parses apr's Assistant: transcript. apr's chat cells carry backend_verified: false, because apr chat reports no backend (apr chat --no-gpu still uploads the GGUF to CUDA (VRAM held outside the GPU lock), and apr chat reports no machine-readable backend #3794); not_covered names it. Run-verb token parity isn't claimed for chat.

Why a pty

Measured on lambda:

  • fed a pipe, llama-cli loops on empty > prompts until killed;
  • ollama run reads the whole pipe as one prompt and answers both turns in one reply.

Neither takes turns from a pipe, so a piped "multi-turn" test of either would have been a single-turn test wearing a label.

Found while building it: #3794

apr chat --no-gpu still uploads the GGUF to CUDA: it holds VRAM outside the GPU lock, while generation does honour --no-gpu. And apr chat reports no machine-readable backend.

Verified

  • bash scripts/check_crux_inference_judge.sh: 42 ok, 0 broke. The chat rows are GREEN, RED (apr "9" vs llama.cpp "12"), RED (no Assistant turn), and the turn/backend/parity record. Three chat mutants each break a row.
  • A real run on lambda (cpu lane, qwen2.5-coder-1.5b Q4_K_M, --verbs run,chat): PASS, 4 GREEN. In the chat cell apr, llama.cpp@d1d3c3396 and ollama 0.33.2 each gave ["2 + 2 equals 4.", "4 * 3 equals 12."], and no ollama import was left behind.
  • bash scripts/guard_tree.sh --no-cargo: 77 checks, 0 failed on 05685a2de. check_no_competing_harnesses.sh: count=0.

🤖 Generated with Claude Code

…at via stdin, llama.cpp and ollama via a pty driver (neither takes turns from a pipe), judged on the final turn

Refs #3795

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Contributor Author

Every commit is already in release/0.69.1-batch-2 (git cherry: 0 unfolded). It reaches main through the release batch PR #3835. Closing as folded, not rejected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant