Skip to content

check runs every check twice per run, and --strict ignores --severity #5

Description

@royalpinto007

What is wrong

cmd_check runs every check over every run twice.

agentrace/cli.py line 71, inside the per-run loop:

findings = analyse(r)

then again at line 90, after the loop:

has_high = any(f.severity == "high" for r in runs for f in analyse(r))

analyse (in agentrace/checks.py) executes all seven checks, several of which run re.finditer over the full result text (check_unverified_claim, check_url_without_verification). On a real session, the README cites 152 runs with 257,721 result characters, so this doubles the regex work for a value that was already available.

There is also a subtle behaviour question worth settling in the same change: the --severity filter at line 73 narrows what is printed, but line 90 ignores it. So agentrace check --severity low --strict can still exit 1 because of a high finding the user explicitly filtered out.

Steps

  1. In cmd_check, compute the findings once. Keep a running flag or collect the per-run findings into a list before the --severity filter is applied.
  2. Replace line 90 with a check against that value, so analyse is called exactly once per run.
  3. Decide what --strict should mean when --severity is given, and make the code and the --strict help text at line 168 agree. Exiting on findings the user asked not to see is the surprising option.
  4. Optional but welcome: a test in tests/test_agentrace.py that calls agentrace.cli.main(["--file", str(fixture), "check", "--strict"]) against tests/fixtures/session.jsonl and asserts the exit code.

Contained to one function, no new dependencies. Comment below to claim it; a reply usually comes within a day.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions