Evidence-gated rearchitecture: measured retrieval/synthesis overhaul, eval harness, code-review fix-set - #951
Open
PromtEngineer wants to merge 30 commits into
Open
Evidence-gated rearchitecture: measured retrieval/synthesis overhaul, eval harness, code-review fix-set#951PromtEngineer wants to merge 30 commits into
PromtEngineer wants to merge 30 commits into
Conversation
Full rebuild driven by a verified audit of ~180 doc/code discrepancies and 52 dead-code findings, then hardened by a browser-driven end-to-end test and an evidence-gated adoption process (see eval/ and Documentation/research/ in the follow-up commits). Core rearchitecture: - Single RAG API server (api_server_with_progress.py deleted); factory.py is the only factory; main.py is master config + a thin working CLI - Backend gateway reads RAG_API_URL; every documented env var is actually read (GENERATION_MODEL, ENRICHMENT_MODEL, EMBEDDING_MODEL, RERANKER_MODEL, LANCEDB_PATH, DB_PATH, NEXT_PUBLIC_*); Docker wiring fixed end to end - Hybrid retrieval = LanceDB FTS + dense fused by RRF; retrieval_mode honored; no-op knobs (denseWeight, chunkOverlap, BM25 config, vision path) removed - Streamed turns persisted via POST /sessions/<id>/messages/save with sources and the pipeline step cascade in message metadata; backend is the sole writer of chat rows - Failed document conversion is a hard error, never a silent empty index; index status advances created->built; thinking-model JSON calls fixed (top-level think:false — chat_template_kwargs is ignored by /api/generate) Evidence-gated defaults (measurements in eval/DECISIONS.md): - Embedder: microsoft/harrier-oss-v1-0.6b with query-side instruction prefix (mixed-corpus first-stage nDCG@10 0.915 vs 0.875 for the 8GB Qwen3-4B) - Default profile reranking OFF: bge-reranker-v2-m3 measured net-negative on this first stage; Qwen3-Reranker-4B (0.977) is the lazy opt-in via a custom scorer (rerankers 0.10.0 silently mis-scores Qwen3 rerankers) - Per-table embedder-identity markers + L2 normalization (text_pages_v4); same-width model swaps now refuse instead of corrupting - Gateway routing is a deterministic gate (~750ms/message saved; 155/155 regression tests); evidence-sufficiency retrieval retry; decomposition moved to the rerank stage; VERIFIER_MODEL seam; graph module removed Dead code removed throughout (frontend components, dup requirements, broken scripts); all Documentation/ rewritten to describe only shipped behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cisions Phase 0 of Documentation/research_roadmap.md — the gate everything else was measured against: - eval/goldset: 72 reverse-generated queries over three corpora (two planted-fact PDFs + Documentation/*.md), labeled by answer-bearing text so they survive re-chunking and embedder swaps; per-row verification records - eval/run_eval.py: in-process recall@5/10/20 + nDCG@10 through the shipped RetrievalPipeline path (first stage, retry, rerank), deterministic, with --embedder/--reranker/--retry/--decompose flags for A/Bs - eval/judge.py: binary groundedness judge validated on 20 hand-built cases - eval/smoke_e2e.py: scripted end-to-end (servers, index, chat, persistence round-trip), 25/25 assertions, clean teardown - eval/BASELINE.md + eval/DECISIONS.md + eval/decisions/*: every measurement, every adoption call, and the gate corrections — including the three cases where this harness overrode the published literature Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Documentation/research/: three primary-source sweeps of agentic-retrieval SOTA as of Aug 2026 (industry, academic, component map), each claim graded established/emerging/contested with rejection logs and could-not-verify appendices. Evidence about the field, not claims about this repo. - Documentation/research_roadmap.md: the phased, eval-gated plan that turned that evidence into the shipped defaults; Phases 0-3 complete with honest outcome notes (including retracted/corrected evidence claims) - Documentation/design_rationale.md: per-component what/why/what-would-change, citing evidence and eval numbers, plus the deliberately-not-implemented list Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rch ideas Six mechanisms from PromtEngineer/agentic-file-search mapped onto the retrieval-first architecture as escalation tiers (full-document escalation, cross-reference hop, overview prefilter, metadata filter DSL, token tracking, ephemeral ask-a-folder), with the loop/scan patterns the 2026 evidence deprecates explicitly excluded. All items flag-gated and benchmark-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…asurement Implemented behind flags, benchmarked on/off, and set per the numbers (evidence: eval/decisions/phase4-*.md): - 4.5 per-query token tracking (ON): ContextVar TokenUsageTracker in ollama_client; token_usage per stage on /chat and the SSE complete event - 4.4 metadata filter DSL (no flag, inert unless a filters object is sent): rag_system/retrieval/filters.py compiles JSON filters to LanceDB where- clauses prefiltering both search legs; quoting characters refused, never escaped; gateway forwards filters and treats one as force-RAG - 4.6 ask-a-folder CLI: ephemeral index, standard pipeline, cleanup verified including on SIGTERM (rag_system/ask_folder.py) - 4.2 crossref extraction (ON, index-inert regex incl. numeric-prefix-stripped filename aliases); query-time hop REJECTED as a default (flag kept, OFF): 0/11 fires at shipped k=20, 0/11 target precision where forced, beaten by raising k at equal context budget - 4.1 full-document escalation (OFF, HOLD): judged lift on fired queries is confounded with the serving-side ~8k silent prompt front-truncation; the one product-default fire regressed - 4.3 overview prefilter (OFF): boost helps only heterogeneous corpora; restrict rejected (removes answer documents from recall@20) Also: api_server --port is honored; escalation subclass hooks documented as load-bearing in retrieval_pipeline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- acquisition corpus (10 M&A PDFs), 24 gold rows (11 requires_crossref), 100 planted facts with 54 cross-references, verify_crossref_goldset.py - run_eval.py: final-candidate-list metrics (recall_final@k, ndcg10_final) alongside first-stage - the crossref hop appends to `documents`, so scoring first_stage alone reports a flat line by construction; invariant check final==first_stage when rerank+hop are off; --crossref-hop / --overview-prefilter / --overviews / --k toggles; hop-precision columns - BASELINE.md: rebuilt-index baselines (post resolver fix) + determinism protocol; zero drift vs tracked numbers, invariant green on all corpora - six Phase-4 decision files: escalation+tokens, crossref+prefilter (with the gate-correction appendix on the resolver fix), filters+ask-folder, the eval-metric wave's zero-hop root-cause, retrieval benchmarks (12 arms + budget-matched controls), answer-quality A/Bs (judged, with the 8k truncation confound and judge-noise findings) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- research_roadmap.md: Phase 4 marked complete with per-item verdict table; 4.4 row corrected (page/date filters did not ship) - design_rationale.md: new 5a (token accounting), 5b (filters + ask-a-folder), 13a (implemented-but-disabled-by-measurement: escalation HOLD, hop REJECTED as default, prefilter boost HOLD / restrict REJECTED, with the deciding numbers) - api_reference.md: token_usage on /chat and the complete event; filters on both chat request shapes with the validation contract; retrieval_retry / crossref_hop / document_escalation SSE events documented (retrieval_retry was a pre-existing parity gap); steps? on /messages/save - improvement_plan.md: Phase-4 rows graduated to Landed; new backlog: context- window budgeting (verified 8194-token silent front-truncation), query-aware hop targets, FTS sub-query sanitization, page/date columns, token_usage UI + gateway passthrough, judge/smoke nondeterminism notes Gold-set integrity after these edits: 176/176 planted facts verified, all goldset row checks pass, smoke 25/25. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ollama_client never set options.num_ctx, so every call inherited the server's split-slot ceiling (measured 8194 prompt tokens on this host) and Ollama silently dropped the FRONT of oversized prompts — deleting the top-ranked evidence from synthesis prompts with no error. Both clients now request a window sized to the prompt, bucketed 8k/16k/32k with a per-model monotonic ratchet: changing num_ctx forces a KV-cache reallocation, and the agent's alternating small/large calls made naive bucketing 2.5x slower (smoke 924s vs 363s pre-fix); the ratchet only grows the window, running the suite in 208s. A response whose prompt_eval_count fills its window logs a truncation warning. OLLAMA_NUM_CTX pins an exact value; OLLAMA_NUM_CTX_MAX caps the bucket (default 32768). Verified: the 99k-char probe that returned prompt_eval_count 8194 and lost a fact planted at position 0 now evaluates 17,351 tokens and recovers it; smoke 25/25 (one known q4 temp-1.0 flake mid-sequence, rerun green). Unblocks the roadmap 4.1 escalation re-run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Executes the condition phase4-answer-quality.md §9 set for the HOLD: re-run the exact A/B once prompts fit. They fit (249 calls, zero truncation warnings, max prompt_eval_count 27516/32768) and the lift did not survive — it inverted. The escalation-off baseline on the identical acqdocs fire subset went 0/9 -> 7/9; on fired rows the direction is now 5/7 -> 2/7 mechanically and 5/7 -> 5/7 by hand adjudication (the four mechanical HARM rows are judge artifacts whose verdicts contradict their own reasons). Both fires under the true product default were regressions on both dates. The 2026-08-09 lift was front-truncation favoring the tail-appended document, exactly as §6.2 hypothesized. Verdict: 4.1 REJECTED as a shipped default (was HOLD); flag and code kept. New decision record eval/decisions/phase4-escalation-rerun.md (agent-run, gate-validated: all cells recounted from raw JSONL); §4 of phase4-answer-quality.md marked VOID (its 0/9 baseline was a truncated-context artifact); verdict tables updated in research_roadmap.md, design_rationale.md §13a, improvement_plan.md. New backlog: judge verdicts contradicting their own reasons (strip the verifier suffix before judging), and a 35k-char verbatim-transcription failure mode when whole documents enter an untruncated window. Gold-set integrity re-verified after the doc edits: 176/176 facts, all acquisition row-level checks pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rd-case benchmark
The Phase-4 A/Bs showed the qwen3.5:4b judge returning verdicts its own
reasons contradict on exactly the rows that decide feature adoption, with
one reason citing the verifier's appended "[Confidence: N%]" suffix as
grounds for rejection (eval/decisions/phase4-escalation-rerun.md §6).
Three changes, all eval-only (the serving path stays fully local):
1. judge.py strips the verifier suffix from both judge slots before
scoring. Measured effect on the 18 hand-adjudicated hard rows: the 4b
goes 13/18 (k=5, unstripped) -> 15/18 (k=1, stripped) — the suffix was
a real perturbation source, fixing 3 of the 4b's 5 known-wrong rows.
2. GroundednessJudge routes any model named claude-* through the
anthropic SDK (lazy import; JSON shape enforced server-side via
output_config json_schema; refusal-aware). Select per run with
JUDGE_MODEL=claude-sonnet-5. Not validated live yet — no API
credentials on this machine; validation commands are in the
validate_judge_hard.py docstring.
3. The re-run's 18 manually-labeled rows are preserved as a permanent
judge benchmark: eval/judge_hard_cases.jsonl (answers + gold + labels
+ the 4b's recorded votes) and eval/validate_judge_hard.py (k-vote
scorer; exits nonzero unless the candidate beats the 13/18 baseline).
Phase-0 20-case validation re-run after the change: 19/20 (0.95, gate
>=0.90). The one flip is unrelated to the edit — suffix stripping is a
no-op on that case's text; it is the documented temp-1.0 paraphrase
noise ("indicates pump cavitation" vs "Pump cavitation detected").
Gold set re-verified after the doc edit: 176/176.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…plits No API key is available on this machine, so validation ran through three independent Claude Sonnet subagents instead of the API backend: each voter executed the exact v1 judge prompt (verifier suffix stripped, byte-identical to what eval/judge.py sends) over all 38 cases in isolation. Result: 20/20 on the Phase-0 validation set (TPR 1.0, TNR 1.0) and 18/18 on the hard-case set, with zero non-unanimous rows across 114 judgments. That includes all 5 rows the qwen3.5:4b k=5 majority got wrong, and the fact-present-but-prefaced-with-a-denial answer (acq_q09) that even the suffix-stripped 4b failed in both arms. Verdict reasons were spot-checked for substantive evidence citation (not leakage; voters never read labels). Baselines on the same hard set: 4b unstripped k=5 = 13/18; 4b stripped k=1 = 15/18; Sonnet subagents k=3 = 18/18. Record: eval/decisions/judge-sonnet-validation-2026-08-13.json (per-case votes). improvement_plan.md documents the protocol: direction-deciding cells use 3 Sonnet subagent voters, majority decides; the 4b stays as the free bulk-pass judge. Gold set re-verified after the doc edit: 176/176. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…back
Two known product bugs from the Phase-4 backlog (improvement_plan §1.7, §6):
1. The streaming synthesis paths never set think, so the 9b generation
model defaulted to thinking and could burn its context window on
chain-of-thought that never enters `response`, returning an empty
answer (measured: prompt 9351 + thinking 7033 = window exactly).
All three sites — RetrievalPipeline._synthesize_final_answer and both
agent-loop compose streams — now pass enable_thinking=False, verified
at the wire (think:false present on the captured Ollama payload).
Side effect: the E2E smoke suite dropped from 208s to 60s, since
every synthesis call was previously paying for hidden CoT tokens.
2. A decomposer sub-query wrapped in double quotes made the LanceDB FTS
parser raise ("position is not found but required for phrase
queries") and the whole hybrid retrieve() returned nothing instead
of using the healthy dense leg (1/24 gold queries in the Phase-4
A/B). retrievers.py now strips double quotes before the FTS leg, and
a hybrid FTS-leg failure of any kind degrades to dense-only with a
warning; fts_only mode still propagates. Verified against the exact
incident query from phase4-answer-quality.md §3.2: 0 docs -> 5 docs
with the answer document ranked first; degradation exercised with a
simulated FTS failure; filtered-search behavior unchanged.
Smoke: 25/25 (60.2s wall).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MarkdownRecursiveChunker._split_text reassembled re.split's captured separators with a loop that consumed two elements but advanced three, silently discarding the first and every alternate body segment whenever a document exceeded max_chunk_size on a separator pass. Long documents entered the index at ~50% content: RFC 9000 retained 49.12%, design_rationale.md 49%. The authored eval corpora (tiny acq PDFs, mostly-small docs files) never made the loss visible; the unseen-corpus RFC shakedown did — 10 of 24 gold answers were absent from the index. The loop now keeps segment 0 and reattaches each separator to the segment that follows it. Verified: 30 randomized structured documents round-trip with zero non-whitespace loss through the split stage; the 6-section synthetic repro retains all sections (was dropping 2); RFC corpus worst-file retention 94.3% and design_rationale.md 99.8%, with the remainder being whitespace normalization only. Smoke 25/25. Consequence recorded in improvement_plan.md: every index built before this fix under-contains its long documents and should be rebuilt; pre-fix docs-corpus baselines are not comparable to post-fix numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… + gold set First eval corpus not authored by us: 23 IETF RFCs (QUIC/HTTP-3 family plus dependencies), 1.44 MiB, 110 directed intra-corpus references, every document connected to >=4 others. Downloaded verbatim from rfc-editor.org; MANIFEST.md records each file's URL and role; download.py re-fetches and verifies the corpus (--check validates the reference graph). Gold set: 24 hand-authored rows (16 factoid / 3 negative / 3 procedural / 2 comparative; 10 requires_crossref, 2 multi-document), every expected string mechanically verified verbatim-present in its named source by eval/verify_rfc_goldset.py (26/26 fact checks, 24/24 flag consistency; gates re-run at the gate independently). This corpus is what exposed the chunker data-loss bug (7d71051) and the crossref extractor's 0/1403 resolution rate on naming conventions we did not author. Shakedown results land in a follow-up commit once the Sonnet judge panel completes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sis is the bottleneck Post-chunker-fix results on the 23-RFC unseen corpus, gate-validated: - Index: 683 chunks (was 387 — the recovered half), 51.6 min with enrichment, zero truncation warnings. Gold reachability 24/24 (was 14/24). - Retrieval (--retry off): recall@5/10/20 = 0.750/0.833/0.958, nDCG@10 0.659 (crossref slice 1.000 recall@20 / 0.719 nDCG). Live gate reproduction matched recall@20 exactly and the rest within one query. Versus the authored corpora (0.958 R@10 / 0.738 nDCG) unseen-real is harder by a same-domain-distractor margin, not broken. Retry fires 1/24. - E2E answers, judged by the validated Sonnet subagent panel (3 voters, all 24 rows unanimous): 5/24 contain the gold fact — single-doc 1/14, crossref 4/10. The 4b k=5 agreed 22/24; both disagreements were rows it had self-flagged as suspect. - Crossref extraction: 0/1403 resolved on real RFC naming — filenames absent from prose, bracketed section citations discarded by _SECTION_RE, no RFC-NNNN pattern. Filed as item 1.8. - New bottleneck filed as item 1.9: with retrieval fixed, the 9b synthesizes from its prior on dense technical text and fabricates citations (e.g. a invented quote from "RFC 9002 §13.4" while the correct value sat in the retrieved snippets); the verifier flags low confidence but the wrong claim still leads. Full record: eval/decisions/rfc-shakedown-2026-08-13.md (+ per-row Sonnet votes). Gold set re-verified after doc edits: 176/176. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… reject 35b Five-arm A/B on the unseen 24-question RFC corpus, all arms judged by the validated Sonnet subagent panel (3 blind voters per arm); full record in eval/decisions/synthesis-grounding-ab-2026-08-13.md: A shipped prompt, qwen3.5:9b 5/24 (baseline) B strict prompt, qwen3.5:9b 6/24 C strict prompt, qwen3.5:9b, temp 0 7/24 (zero judge splits) D strict prompt, qwen3.6:35b-a3b 4/24 D2 shipped prompt, qwen3.6:35b-a3b 5/24 Landed (arm C's exact tested config): the synthesis prompt loses its "General knowledge" escape hatch — the hole every observed fabricated-citation failure went through — and forbids quotes, section numbers and document names not present in the snippets; synthesis and the agent's compose stream decode at temperature 0 (stream_completion gained an `options` kwarg, wire-verified). Stated honestly: the +2 is at the harness noise floor and is not claimed as a measured quality win — adoption rests on escape-hatch removal, decode determinism (targets the documented temp-1.0 smoke/eval flake), and zero cost. Bigger model REJECTED: qwen3.6:35b-a3b measures equal-or-worse under both prompts — grounding on dense unseen text is not parameter count at this scale. (It is ~2x faster per query and stays a viable GENERATION_MODEL env swap for speed on large-RAM machines.) Item 1.9 stays open; the identified next lever is the compose gap (the composer writes the judged answer on 4-9/24 rows and carries no grounding rules; the mandated abstain sentence was emitted 0 times in 120 answers). Smoke 25/25 (first attempt failed on an unrelated stray `python -m http.server 8000` squatting the gateway port; killed). Gold set re-verified after doc edits: 176/176. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…p arm C Single-variable test of the compose-gap hypothesis: the composer prompt got the same hard grounding rules as synthesis. Sonnet panel: 4/24 vs arm C's 7/24. Attribution is clean — compose fired on 8/24 rows and all three pass->fail flips vs C were composed rows (0 gains among them); the strict copy-contract that works against raw snippets drops facts when applied to already-synthesized sub-answer prose. Change reverted; shipped config remains exactly arm C. Decision-file appendix records the negative result and the remaining 1.9 levers (abstain-on-low- verifier-confidence, deterministic decomposition, passage citations). Gold set re-verified after doc edit: 176/176. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause, measured: synthesis prompts were ~94k tokens ("512-token"
chunks store at ~823 tokens with enrichment prefixes; two retrieval legs
returned 20+20 with no dedupe; the +-1 sibling merge tripled each entry)
while Ollama's parallel-slot split served ~16k and silently
front-truncated — deleting the TOP-ranked evidence on every call.
- retrieval_pipeline: dedupe across base + late-chunk legs on
(document_id, chunk_index); new _budget_synthesis_context() packs
rank-ordered docs into an explicit token budget (default 12k,
synthesis_context_tokens config) with sibling-span overlap
suppression and a min-one-doc guarantee.
- ollama_client: slot-proof front-truncation warning
(prompt_eval_count < prompt_chars // 6 catches the served-window
split the num_ctx comparison was blind to).
- .env.example: OLLAMA_NUM_PARALLEL=1 note.
Arm F E2E on the unseen RFC corpus (3-voter blind Sonnet panel,
2-of-3 majority, verifier suffix stripped): 16/24 judged grounded
(single-doc 10/14, crossref 6/10) vs shipped arm C's 7/24 (2/14,
5/10). One split across 72 votes. Mechanics: context 335k -> ~40k
chars, truncation warnings 32 -> 0, cited-expected-source 24/24,
wall time -32%. Per-row votes in synthesis-ab-arm-f-panel.json;
narrative in the decision file's arm F appendix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rm G) With the 12k budget controlling how MUCH context synthesis gets, the reranker now controls WHICH and HOW MANY docs get in — threshold selection, not just reordering (user-directed design): - reranker.min_score (default 0.5): union-of-max across the original query + sub-queries — a candidate survives if the calibrated Qwen scorer marks it relevant to ANY of them. min_keep floors the selection at 3; top_k 10 caps it; the token budget stays as backstop. Threshold applies only to the Qwen scorer class (calibrated P(relevant)); raw cross-encoder logits have no fixed scale. - Candidates are scored on their core chunk text (preserved in metadata.core_text at latechunk-merge time), not the +-1-merged block — the merge buries the matching chunk past the scorer's 2,048-token truncation window. - Default profile: enabled: true with Qwen/Qwen3-Reranker-4B. Supersedes the Phase-1 off-by-default call, which predated the context budget (rank order barely mattered under front-truncation). Arm G E2E on the unseen RFC corpus (3-voter blind Sonnet panel): 18/24 grounded (single-doc 11/14, crossref 7/10) vs arm F's 16/24 (10/14, 6/10); gains q02/q17/q22, loss q20 (2-1 judge-nuance split). Net +2 is within the 1-2 row noise floor — adopted as a no-regression user-directed feature, not a claimed quality win. Selection is genuinely adaptive: mean 8.8 docs kept, range 3-10 (one query kept 3/36). Cost: median query 45s -> 71s (+92% total wall) from the 4B scorer on MPS; opt-out reranker.enabled: false restores arm F. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n shape)
Decomposed queries used to run one full pipeline per sub-query (N rerank
passes, N synthesis calls) and compose the sub-ANSWERS. Now they pool:
- _pooled_first_stage(): per-sub-query first-stage retrieval, pooled and
deduped on (document_id, chunk_index), round-robin interleaved, each
candidate tagged with its source sub-queries.
- One rerank pass scores each candidate ONLY against the sub-queries
that retrieved it (union-of-max, same pair cost as the old per-SQ
passes), with a per-sub-query floor so no sub-question's evidence can
be entirely thresholded out.
- One synthesis against the original question. The composer — where
arm E measured fact loss — drops out of the decomposition path.
Config: query_decomposition.{compose_from_sub_answers: false,
pooled_first_stage: true}.
QueryDecomposer now decodes greedily (temperature 0, via a new options
kwarg on generate_completion; wire-verified): identical splits and
sub-query text across repeat runs, ending the temp-1.0 row-set drift
that had muddied every synthesis A/B.
Measured on the unseen RFC corpus, 3-voter blind Sonnet panels:
first run (sampled decomposition) 17/24 vs arm G's 18/24 with only one
pooling-attributable flip; per user decision, determinism was fixed and
both paths re-run on IDENTICAL row sets (6 decomposed queries, verbatim
same sub-queries): pooled 17/24 vs composer 17/24 — dead tie, zero
split votes across 144 judgments, and all 18 direct-path rows judged
identically across arms. Adopted on structure: 24 vs 32 synthesis
calls, linear (not multiplicative) scaling with sub-query count, one
less failure surface. Panel records + decision-file appendix included.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Diagnosis of the surviving crossref failures showed half were
attribution failures, not fact failures: answers with every salt,
parameter name and identifier character-for-character correct were
judged ungrounded only because they could not name the defining RFC.
Root cause was ours — the synthesis context was a bare join of chunk
texts with no source labels, while strict-prompt rule 3 (correctly)
forbids writing document names not present in the snippets. RFC
corpora cross-reference by tag ("[QUIC-TLS]"), so the model had no
grounded path to "RFC 9001" even though the pipeline knows every
chunk's source file.
Every snippet in the synthesis context now opens with
"[Source document: <document_id>]", and new prompt rule 7 tells the
model to attribute facts via those labels when sourcing matters.
Grounding stays strict: document names are now IN the snippets.
Arm I on the unseen RFC corpus (3-voter blind Sonnet panel, identical
deterministic row set to arm H2): 21/24 grounded (single-doc 12/14,
crossref 9/10) vs H2's 17/24 (11/14, 6/10) — gains q06/q18/q20/q24,
ZERO losses, one split. +4 with no regressions is well outside the
established 1-2 row noise floor. RFC arc: 5 -> 7 -> 16 -> 17-18 -> 21.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…agnosed HEAD (a3f999a) vs pre-tuning baseline (80d5215) on all 96 authored gold rows, identical indexes: acq 16->18 (4b), atlas7 19->19 (4b), docs 15->17 (Sonnet panels, 0 splits), hr 24->21 (Sonnet panels, 0 splits — real). Mechanical exact-substring 24/96 -> 61/96; wall time -27%. hr losses diagnosed in-process: evidence WAS in context (2-chunk corpus, both kept); the strict prompt answers the literal question and omits adjacent clauses the golds include — style, not grounding. Verdict: not overfit; changes stay; completeness-prompt tweak queued with 5-bench validation required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…get rows Rule-8 candidate for the hr -3 regression fails its purpose: hr_h05 answer byte-identical to the panel-rejected HEAD answer, h08/h13 still missing the gold qualifiers. 4b screen movement (+3/96 authored, rfc 14=14 same-judge) is noise-band churn with new qualifier losses (hr_h04, acq 18->16). Strict prompt stays at the arm-I 7-rule form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every _lc table lacked an FTS index (indexing created one only on the base table), so the hybrid lc leg silently ran dense-only on ALL corpora since late-chunking landed. Arm K (120 rows, index present vs absent): E2E-neutral, 5/96 rows changed citations, Sonnet panel 0 regressions on all deciding cells. Also: 4b screen judge sampled at default temp — 11/24 verdict flips on byte-identical answers; GroundednessJudge now decodes at temperature 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n prompt frozen Two-variant prompt selected on bool(chat_history). Multi-turn variant adds a last_assistant_answer field (wrong-entity pronoun resolution measured on the new eval/goldset/multiturn.jsonl: 'their'->StartupXYZ at baseline, MegaCorp after) + ellipsis rule 1b (follow-ups no longer collapse into the previous question). Single-turn variant stays byte-exact: arm L measured m1c's prompt additions costing 2 Sonnet-confirmed rfc rows via degraded decompositions; verified byte-identical temp-0 decompositions on all 120 single-turn gold queries after the split. Multi-turn E2E 12/12 (arm m1d). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings addressed (all verified before commit): - docling_chunker: split_markdown infinite loop -> index-based window with strict progress; element walk rewritten against real docling_core API (label.value/section_header/prov) — heading paths and reading order restored - agent/loop: per-request overrides now snapshot/restore shared config (was sticky process-wide); triage pinned temp 0; history capped 40 turns with verifier tags stripped - UI defaults realigned to shipped profile (aiRerank on, compose off) — arm G-L quality now actually reaches UI users - retrievers: FTS score read from _score; retrieval_pipeline: refuse-dont- escape document_id in context expansion, filter passed into executor workers (thread-local not inherited), per-call table name - embedders: delete-by-document_id before append (rebuilds no longer duplicate); latechunk: sliding windows over full doc (tail chunks got CLS-token garbage past 8192 tokens) - backend: sqlite _connect() factory with FK pragma + 30s timeout; upload size cap; chat errors return 502/504 (was 200-with-error-text); chat_data.db untracked - verifier: grounded/confidence coercion fail-open; verifier+retry+async client temp-pinned (options param added to generate_completion_async) - eval: anthropic judge temp 0; run_eval realigned to arm-G profile with index/prompt cache versioning; smoke pre-flight port guard; rfc corpus registered - frontend: session index-state clearing, regenerate dedupe, SSE error handling, quick-chat persistence, memoized markdown, tsc gate re-enabled - dead code removed (legacy prompt blocks, unused endpoints/components, duplicate compose file); .github/workflows/ci.yml added (155-case gateway routing suite; passing) Single-turn decomposer prompt verified byte-identical on all 120 gold queries post-change (decomp stability gate). Synthesis + judge prompts untouched. Index-affecting fixes require bench-index rebuild; measured in the follow-up eval run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…odes, line refs) Propagates the two landed default changes (reranker ON with threshold selection since arm G; pooled decomposition replacing compose since arm H) across README, system/architecture overviews, design_rationale, api_reference, deployment/docker/quick-start guides, and eval docs. Fixes the specific falsehoods the docs audit flagged (502/504 now true after the code fix, lazy reranker load, per-chunk enrichment, prompt_inventory ghosts) and drops hard line-number references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… recovered Full 120-row rerun on rebuilt v2 indexes (fix-set changed index content: acq 13->82 chunks etc). Deterministic 4b: authored 74->78. Full Sonnet panel on rfc arm M: 19/24 vs arm-I 21/24 — q06/q24 real losses, q10/q15/ q20 real gains from the corrected latechunk vectors. hr_h05 (unfixable by prompt rules) recovered via proper chunking. Multiturn 12/12 held. Fix-set stays: correctness over bench-row protection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…treaming Streaming 'stopped working' root cause: the Next dev watcher shares its root with the Python backend, so every chat_data.db write Fast-Refreshed the page mid-stream, remounting React and aborting the in-flight SSE fetch (net::ERR_ABORTED). next.config.ts now scopes the watcher to frontend sources. Verified: no rebuild storms, live DOM growth during both quick-chat and session-chat generations. Avatars: AvatarFallback rendered black text on the dark bg-muted circle — invisible once the third-party pravatar image was removed. ChatBubbleAvatar fallbacks now use an explicit light scheme; user bubble shows 'U'. Quick Chat now streams: new gateway POST /chat/stream (SSE, same token/complete/error framing as the RAG API so the client parser is shared), chat_stream() on the backend Ollama client, api.streamChatMessage(), and token-by-token rendering in quick-chat.tsx. Gateway suite 155/155; tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Markdown render sat inside a whitespace-pre-wrap container, so each newline in the answer displayed twice — once as ReactMarkdown's paragraph margin and once as a literal line break — producing large vertical gaps around headings and between paragraphs. Fixes: - drop whitespace-pre-wrap from all four Markdown-wrapping sites - add remark-breaks so intentional single-newline breaks still render (the reason pre-wrap was there in the first place) - tighten prose margins for chat bubbles (prose-p:my-2 etc.) - make normalizeWhitespace fence-aware: whitespace inside ``` code blocks is no longer collapsed (it was corrupting code indentation and ASCII tables quoted from documents) Verified live: prose computes white-space normal, real p/li/strong elements, zero 3+-newline runs on the rendered page. tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The complete August 2026 evidence-gated rearchitecture of localGPT: 28 commits in which every behavioral change was gated on measurement — a permanent eval harness with five benchmark corpora, blind 3-voter Claude Sonnet judge panels for direction-deciding cells, and decision records for every adopt/revert call (see
eval/decisions/, 15 records).Headline results
RFC bench (24 hard questions over 23 real IETF RFCs, unseen during development): 5/24 → 19/24 Sonnet-judged, with −14% query latency. Authored benches (acq/atlas7/hr/docs, 96 rows): 78/96 deterministic-judge, up from a 65/96-equivalent baseline. New multi-turn conversation bench: 12/12.
Major changes (each with its decision record)
Retrieval & synthesis pipeline
Bug fixes from an adversarial code review (all verified, then measured)
split_markdowninfinite-loop; docling tree-walk rewritten against the real docling_core API (indexes were structurally collapsed: acq corpus 13→82 chunks after fix)_lctables get FTS indexes (hybrid leg had been silently dense-only everywhere)Infra
eval/: 5 gold sets (120 single-turn rows + 12 multi-turn conversations), judge with validated Sonnet-subagent protocol, smoke test with port-collision guard, decision recordschat_data.dbuntrackedHonest ledger
The final fix-set traded rfc 21→19 (two crossref rows lost to candidate redistribution, three long-standing losses recovered) while authored rose +4 — kept because reverting correctness fixes to protect bench rows is reverse bench-tuning. Known residue, all documented: rfc crossref rows q06/q24 (extractor patterns, plan item 1.8), hr h08/h13 (qualifier-omission style), scanned/OCR corpus eval still missing.
Verification
tsc --noEmitclean (build gate re-enabled); all eval numbers reproducible from committed harnesses (rfc E2E runner adaptation pending — noted ineval/decisions/fixset-impact-2026-08-17.md)🤖 Generated with Claude Code