feat(pr-review): make the review pipeline inspectable - #14
Merged
Conversation
Log the routing decision, analysis settings, every compilation choice, and the publication result; always print the analysis session output; and upload the .pr-review state directory and session transcript as a run artifact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The PR review CI was effectively a black box.
prepare,compose,compile, andpublishprinted nothing but failure annotations, the analysis session's output was hidden behind adebug_logsinput defaulting tofalse, and everything a run produced —review-input.json, both diffs, the raw model output, the compiled payload — lived in.pr-review/on a runner that then disappeared. When a review reached a surprising verdict, or a finding the model reported never appeared on the PR, there was no way to tell why.This adds three inspection surfaces, all derived from the same trace lines so they cannot disagree.
Job log — collapsible groups, in step order:
Review routing(prepare) — mode and why, prior-state source and whether its version still matches, previous and current head, compare status and file count, both diff sizes, how much of the PR conversation was included versus truncated, the paths in scope, and every open prior finding and question handed to the model.Analysis settings(compose) — model, tier, depth, pre-mortem, turn budgets, allowed tools.Model output (raw, before compilation)andCompilation decisions(compile) — one line per model result the compiler accepted, suppressed as a duplicate of an open finding, dropped at the per-severity cap of five, or rerouted to the review body because its line is not commentable, plus each prior finding and question disposition. These drops were previously silent and are the usual reason a finding exists in the model's output but never reaches the PR.Publication(publish) — review ID, inline comments requested versus posted, inline fallback, threads resolved, sticky comment ID.Step summary — routing decision and reason, head range, the model and turn budget the analysis actually ran with, tier/high-risk/pre-mortem flags, and this round's counts (findings split inline versus review body, new questions, prior findings resolved), then folds in the pipeline trace and the raw pre-compilation model output. A failed run gets the same trace, showing how far the round got.
Run artifact
pr-review-state-<pr>-<run>-<attempt>(14 days by default) carries the bytes themselves: routing input, both diffs,analysis-transcript.json(the session's turn-by-turn record, plus a retry twin), raw and parsed structured output, compiled payload, publish result, andtrace.log. Upload needs no new permissions — it uses the Actions runtime token.Also: when the structured output fails to parse, the unparsable bytes are now kept as
model-output.raw. That is exactly the case where they matter, andcompilepreviously died holding the only copy.Interface changes
debug_logsis deprecated and ignored;show_full_outputis now unconditionallytrue. Consumers still passing it are unaffected.state_artifact(defaulttrue) andstate_artifact_retention_days(default14).actions/upload-artifactpinned to043fb46(v7.0.1).Test plan
python3 -m unittest discover -s .github/actions/claude-pr-review -p 'test_*.py'— 101 tests pass (was 85). New coverage: trace emission and persistence, list truncation, the routing explanation including a truncated conversation, compilation traces for accepted/rerouted/capped/deduped findings and truncated questions, summary detail blocks and their size cap, the analysis-settings rows, and the enriched success report.actionlinton the workflow,bash -non every composite run block, andyqparse ofaction.yml.pr-review-state-*artifact.