Skip to content

Scope an issue's assertions to particular targets #136

Description

@gaurav

The problem

An issue's embedded assertions run against every --target, and there is no way to say
which environments an assertion is about.

tests/github_issues/test_github_issues.py::test_github_issue takes both target_info
(parametrized over the --target list by the root tests/conftest.py) and
github_issue_id (parametrized in tests/github_issues/conftest.py), so the run is a full
cross product. .github/workflows/dashboard.yaml loops over every target in targets.ini,
so an assertion filed today is evaluated daily against prod, test, ci, ci-es, dev and exp.

Combined with the open-issue semantics — an open issue whose assertions all pass is a
strict XPASS, i.e. a reported failure meaning "this looks closeable" — this makes
environment-specific findings unfileable:

  • An assertion describing a bug present only on ci-es fails there (correct, reports as
    XFAIL) and passes on all five other targets, where it then reports a strict XPASS
    failure every day
    until the issue is closed.
  • The reverse is just as bad: an assertion describing something only ci-es gets right
    reports XPASS on ci-es and failures everywhere else.

We have just hit this filing the NameRes ES validation findings, and had to file them as
prose and curl reproducers with no assertions at all — which is the outcome this harness
exists to avoid.

Sketch of a fix

Let an assertion set name the targets it applies to, and skip it on the others. Something
like a targets: key alongside the assertion names in the fenced YAML block, defaulting to
"all targets" so every existing issue keeps its current behaviour. The values would need
validating against the sections in targets.ini — an unknown target name should be a hard
error like an unknown assertion name is, not a silent skip, or a typo turns into an
assertion that never runs anywhere.

The inline {{BabelTest|... marker syntax has no room for a modifier, so this may be
YAML-block-only, which seems acceptable.

Worth deciding at the same time: if every assertion in an issue is skipped for a target,
that should probably be a skip rather than the "produced no test results — check assertion
configuration" xfail the code currently yields.

Not blocked on this

SearchByNameTopResult and DoesNotSearchByName have landed and work; they are simply
unused by any filed issue until this exists.


The NameRes ES validation set: #131 (alphanumeric tokens split) · #132 (ranking:
longer names win) · #133 (score ties push the concept out) · #134 (where ES is better) ·
#135 (/status reports zeros) · #136 (scoping assertions to targets, which is what stops
these being filed with live assertions).

Produced by pytest --target ci-es -n auto tests/nameres/test_nameres_from_gsheet.py
compared against the same run with --target ci. Over the same 2076 Babel Validation sheet
rows: 58 exact top-1 hits on ES against 127 on Solr.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions