fix(preprocess): start the first preprocess run instead of crashing on 404 - #71
Merged
Conversation
…hen the table has no status record yet (404)
SimoneBottoni
requested changes
Jul 9, 2026
SimoneBottoni
self-requested a review
July 9, 2026 16:12
SimoneBottoni
approved these changes
Jul 9, 2026
aural-psynapse
added a commit
that referenced
this pull request
Jul 17, 2026
## Changes **Docs** - **AGENTS.md** — note it is the canonical agent entry point (mirrored as the provably.ai machine view), and make explicit that claims must come from the agent's own `SourceryKitAgentResponse` output, not be hand-assembled. - **docs/handoff.md** — add a `SourceryKitAgentResponse` section describing the typed output and its `ClaimedValue` fields (`path`, `value`, `sourcerykit_ref`), ordered type → component. Correct the `claimed_value` payload row from `Any` to `list[ClaimedValue]`. - **cookbooks/openai_agents** — accept `OPENROUTER_API_KEY` as well as `MODEL_API_KEY`, so a verbatim copy authenticates against an OpenAI-compatible gateway (OpenRouter). **Code** - **evaluate_handoff / build_handoff_payload** — when a claim can't be resolved it was dropped with only a log line, so a payload that ended up with zero claims surfaced as `ERROR` with an empty `errors` list. The drop reason now travels on the payload (`build_errors`) and `evaluate_handoff` returns it in `errors`, so a zero-claim result explains itself. ## Why These changes come out of a discoverability loop that runs a fresh agent from provably.ai to a working SourceryKit integration and turns each observed failure into one fix. Runs kept failing at the same points: agents couldn't find the entry doc, hand-assembled claims instead of binding `SourceryKitAgentResponse`, couldn't authenticate the cookbook with the key they had, and hit an `ERROR` with no reason when claims failed to resolve. Each item above addresses one of those. (The separate preprocess bug that blocked verification end-to-end is fixed in #71.) --------- Co-authored-by: SimoneBottoni <simone.bottoni94@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
run_preprocess()turns a freshly-ingested intercept table into the preprocessed form its collection needs before any query can run or be proven. It checks the current preprocess status first and only then starts a run.The problem: a table that has never been preprocessed has no status record yet, so the status endpoint returns 404. That 404 raised straight out of the initial status check and crashed
run_preprocess()before it ever calledstart_preprocess()— so the very first preprocess never happened.Consequence: the collection stayed
is_valid: false(invalid_reason: table_not_preprocessed), and every proof query against it failed —evaluate_handoff()returnedERRORwith an emptyper_claim. On a fresh setup, verification could never succeed.Changes
run_preprocess(), treat a 404 from the initial status check as "not started" and proceed to start the first preprocess run. Any non-404 error still propagates unchanged.Verification
Reproduced against the live backend: on a never-preprocessed table, row-returning queries returned 500 while
COUNT(*)succeeded, and the collection reportedtable_not_preprocessed. Runningstart_preprocessfirst flipped the collection tois_valid: trueand the same row query resolved. With the fix, a full end-to-end run reachesPASSon an honest claim andCAUGHTon a tampered one.