rivet coverage --tests finds // rivet: verifies REQ-X in .rs and (per the documented syntax) # rivet: verifies REQ-X in Python, but does not read the same # comment from a shell script.
Reproduction
Two files in one directory, one marker each:
/tmp/scantest/src/lib.rs // rivet: verifies REQ-NOKEYDISK-001
/tmp/scantest/src/g.sh # rivet: verifies REQ-INSTALLSHADOW-001
$ rivet coverage --tests --scan-paths /tmp/scantest/src
REQ-NOKEYDISK-001 1 test marker
REQ-INSTALLSHADOW-001 No test markers found
Same path, same comment convention, different extension. So this is the file type, not --scan-paths.
Why it costs us something real
Not every falsifiable check is a unit test, and the ones that are not tend to be the ones guarding the nastiest failures. Two current examples from varve, both of which have a negative control proving they can go red:
tools/no-key-on-disk.sh --self-test — refuses any workflow that redirects key material into a file, and proves it by rejecting four shapes (including the exact line our own deposit workflow shipped for every layer it ever published) while accepting the two documented file-descriptor forms.tools/systest/install-shadow.sh — asserts the installer warns when a different varve wins the PATH lookup, and that it stays quiet otherwise. It also fails if the check is removed from install.sh.
Both run in CI. Both are the sole evidence for their requirement. Neither can be expressed as a marker, so rivet verify refuses:
error: refusing to verify 'REQ-NOKEYDISK-001': no verifying evidence.
Add an incoming `verifies` link from a test/verification artifact, or a
`// rivet: verifies REQ-NOKEYDISK-001` marker in a test, then re-run.
That refusal is correct behaviour on the information rivet has — we are not asking for it to be relaxed. The requirement is genuinely gate-verified, and the only reason it cannot say so is the extension.
The workaround is to leave such requirements at implemented forever, which quietly teaches the reverse of what the traceability gate is for: it makes "verified by an executable check with a negative control" indistinguishable from "not verified", and the difference is exactly what an assessor asks about.
Ask
Read # markers from shell scripts — .sh, and ideally extensionless files carrying a #! shell shebang, since a lot of gates have no extension.
If the concern is that any # comment file would become a "test", a narrower version works for us: only count a shell file when it is under a configured scan path, which --scan-paths already provides.
Happy to send the PR if you would rather have it than the issue — say which.
rivet coverage --testsfinds// rivet: verifies REQ-Xin.rsand (per the documented syntax)# rivet: verifies REQ-Xin Python, but does not read the same#comment from a shell script.Reproduction
Two files in one directory, one marker each:
Same path, same comment convention, different extension. So this is the file type, not
--scan-paths.Why it costs us something real
Not every falsifiable check is a unit test, and the ones that are not tend to be the ones guarding the nastiest failures. Two current examples from varve, both of which have a negative control proving they can go red:
tools/no-key-on-disk.sh --self-test— refuses any workflow that redirects key material into a file, and proves it by rejecting four shapes (including the exact line our own deposit workflow shipped for every layer it ever published) while accepting the two documented file-descriptor forms.tools/systest/install-shadow.sh— asserts the installer warns when a different varve wins the PATH lookup, and that it stays quiet otherwise. It also fails if the check is removed frominstall.sh.Both run in CI. Both are the sole evidence for their requirement. Neither can be expressed as a marker, so
rivet verifyrefuses:That refusal is correct behaviour on the information rivet has — we are not asking for it to be relaxed. The requirement is genuinely gate-verified, and the only reason it cannot say so is the extension.
The workaround is to leave such requirements at
implementedforever, which quietly teaches the reverse of what the traceability gate is for: it makes "verified by an executable check with a negative control" indistinguishable from "not verified", and the difference is exactly what an assessor asks about.Ask
Read
#markers from shell scripts —.sh, and ideally extensionless files carrying a#!shell shebang, since a lot of gates have no extension.If the concern is that any
#comment file would become a "test", a narrower version works for us: only count a shell file when it is under a configured scan path, which--scan-pathsalready provides.Happy to send the PR if you would rather have it than the issue — say which.