The problem
An issue's embedded assertions run against every --target, and there is no way to say
which environments an assertion is about.
tests/github_issues/test_github_issues.py::test_github_issue takes both target_info
(parametrized over the --target list by the root tests/conftest.py) and
github_issue_id (parametrized in tests/github_issues/conftest.py), so the run is a full
cross product. .github/workflows/dashboard.yaml loops over every target in targets.ini,
so an assertion filed today is evaluated daily against prod, test, ci, ci-es, dev and exp.
Combined with the open-issue semantics — an open issue whose assertions all pass is a
strict XPASS, i.e. a reported failure meaning "this looks closeable" — this makes
environment-specific findings unfileable:
- An assertion describing a bug present only on
ci-es fails there (correct, reports as
XFAIL) and passes on all five other targets, where it then reports a strict XPASS
failure every day until the issue is closed.
- The reverse is just as bad: an assertion describing something only
ci-es gets right
reports XPASS on ci-es and failures everywhere else.
We have just hit this filing the NameRes ES validation findings, and had to file them as
prose and curl reproducers with no assertions at all — which is the outcome this harness
exists to avoid.
Sketch of a fix
Let an assertion set name the targets it applies to, and skip it on the others. Something
like a targets: key alongside the assertion names in the fenced YAML block, defaulting to
"all targets" so every existing issue keeps its current behaviour. The values would need
validating against the sections in targets.ini — an unknown target name should be a hard
error like an unknown assertion name is, not a silent skip, or a typo turns into an
assertion that never runs anywhere.
The inline {{BabelTest|... marker syntax has no room for a modifier, so this may be
YAML-block-only, which seems acceptable.
Worth deciding at the same time: if every assertion in an issue is skipped for a target,
that should probably be a skip rather than the "produced no test results — check assertion
configuration" xfail the code currently yields.
Not blocked on this
SearchByNameTopResult and DoesNotSearchByName have landed and work; they are simply
unused by any filed issue until this exists.
The NameRes ES validation set: #131 (alphanumeric tokens split) · #132 (ranking:
longer names win) · #133 (score ties push the concept out) · #134 (where ES is better) ·
#135 (/status reports zeros) · #136 (scoping assertions to targets, which is what stops
these being filed with live assertions).
Produced by pytest --target ci-es -n auto tests/nameres/test_nameres_from_gsheet.py
compared against the same run with --target ci. Over the same 2076 Babel Validation sheet
rows: 58 exact top-1 hits on ES against 127 on Solr.
The problem
An issue's embedded assertions run against every
--target, and there is no way to saywhich environments an assertion is about.
tests/github_issues/test_github_issues.py::test_github_issuetakes bothtarget_info(parametrized over the
--targetlist by the roottests/conftest.py) andgithub_issue_id(parametrized intests/github_issues/conftest.py), so the run is a fullcross product.
.github/workflows/dashboard.yamlloops over every target intargets.ini,so an assertion filed today is evaluated daily against prod, test, ci, ci-es, dev and exp.
Combined with the open-issue semantics — an open issue whose assertions all pass is a
strict XPASS, i.e. a reported failure meaning "this looks closeable" — this makes
environment-specific findings unfileable:
ci-esfails there (correct, reports asXFAIL) and passes on all five other targets, where it then reports a strict XPASS
failure every day until the issue is closed.
ci-esgets rightreports XPASS on
ci-esand failures everywhere else.We have just hit this filing the NameRes ES validation findings, and had to file them as
prose and
curlreproducers with no assertions at all — which is the outcome this harnessexists to avoid.
Sketch of a fix
Let an assertion set name the targets it applies to, and skip it on the others. Something
like a
targets:key alongside the assertion names in the fenced YAML block, defaulting to"all targets" so every existing issue keeps its current behaviour. The values would need
validating against the sections in
targets.ini— an unknown target name should be a harderror like an unknown assertion name is, not a silent skip, or a typo turns into an
assertion that never runs anywhere.
The inline
{{BabelTest|...marker syntax has no room for a modifier, so this may beYAML-block-only, which seems acceptable.
Worth deciding at the same time: if every assertion in an issue is skipped for a target,
that should probably be a skip rather than the "produced no test results — check assertion
configuration" xfail the code currently yields.
Not blocked on this
SearchByNameTopResultandDoesNotSearchByNamehave landed and work; they are simplyunused by any filed issue until this exists.
The NameRes ES validation set: #131 (alphanumeric tokens split) · #132 (ranking:
longer names win) · #133 (score ties push the concept out) · #134 (where ES is better) ·
#135 (
/statusreports zeros) · #136 (scoping assertions to targets, which is what stopsthese being filed with live assertions).
Produced by
pytest --target ci-es -n auto tests/nameres/test_nameres_from_gsheet.pycompared against the same run with
--target ci. Over the same 2076 Babel Validation sheetrows: 58 exact top-1 hits on ES against 127 on Solr.