ci: add Atheris fuzz targets and ClusterFuzzLite#11482
Open
julian-risch wants to merge 6 commits into
Open
Conversation
Address the OpenSSF Scorecard Fuzzing check (0/10) and add real fuzzing of the project's untrusted-input entry points. - test/fuzz/: three Atheris harnesses — Pipeline.loads (serialized pipeline deserialization), Document.from_dict, and document_matches_filter (filter expressions). Each catches the exceptions that are a normal reaction to malformed input so only genuine crashes/hangs/unexpected errors are reported. - .clusterfuzzlite/: Dockerfile + build.sh + project.yaml to build the harnesses with the OSS-Fuzz Python toolchain. - .github/workflows/cflite_pr.yml: short, code-change-scoped ClusterFuzzLite run on PRs that touch fuzzed code, least-privilege token, SHA-pinned actions. - licenserc.toml: exclude .clusterfuzzlite from the license-header check. Scorecard detects this via both the `import atheris` harnesses and the .clusterfuzzlite deployment. pytest does not collect fuzz_*.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Pin gcr.io/oss-fuzz-base/base-builder-python to its current digest instead of the rolling latest tag, for supply-chain integrity. The OSS-Fuzz base-builder is updated frequently, so the comment documents how to refresh the digest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Coverage report
This PR does not seem to contain any modification to coverable code. |
Add a docker ecosystem entry for /.clusterfuzzlite so Dependabot keeps the digest-pinned gcr.io/oss-fuzz-base/base-builder-python in Dockerfile up to date. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- fuzz_pipeline_loads: import DeserializationError from haystack.core.errors (it is not exported by haystack.errors; the old import raised ImportError at harness load, so the fuzzer failed to build). - fuzz_document_from_dict / fuzz_filters: parse raw fuzzer bytes directly with json.loads (accepts bytes) instead of FuzzedDataProvider, so the input domain is JSON text and the seed corpus is usable 1:1. - Add a seed corpus of valid inputs (test/fuzz/corpus/<harness>/) and zip it into <harness>_seed_corpus.zip in build.sh to bootstrap coverage past the JSON parse. - cflite_pr: pass github-token to build_fuzzers so it can check out the PR base that mode: code-change diffs against. - README: document the seed corpus and updated local-run commands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scorecard detection is static (Dockerfile + import atheris) and unaffected by fuzz-seconds; 120s keeps a meaningful per-harness regression pass while trimming CI time on PRs that touch the fuzzed entry points. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`haystack/**` triggered the build-heavy fuzzing job on nearly every library PR, paying the Docker build + install cost before code-change mode found nothing to fuzz. Narrow the path filter to the modules the harnesses actually exercise (core pipeline/serialization/marshal, dataclasses, utils/filters). Deep transitively-reached regressions are left to a future scheduled batch run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
Adds fuzzing of a selection of Haystack's untrusted-input entry points to our CI. There could be more (for example pipeline breakpoint snapshots) but I limited it to the following for now:
Fuzz harnesses
fuzz_pipeline_loads.pyPipeline.loadsfuzz_document_from_dict.pyDocument.from_dictDocumentfrom an untrusted dict.fuzz_filters.pydocument_matches_filterEach harness catches the exceptions that are a normal reaction to malformed input (
DeserializationError,FilterError,ValueError, …) so only genuine crashes, unbounded recursion, hangs, or unexpected exception types are reported. The "expected" lists can be tightened over time to surface subtler bugs.ClusterFuzzLite — .clusterfuzzlite/:
Dockerfile+build.sh+project.yamlbuild the harnesses with the OSS-Fuzz Python toolchain (compile_python_fuzzer).CI — .github/workflows/cflite_pr.yml: a short,
code-change-scoped run (180s) on PRs that touch fuzzed code or the fuzzing setup. Least-privilegecontents: readtoken, SHA-pinned ClusterFuzzLite actions, SARIF upload disabled (so nosecurity-events: writeneeded). Crashes fail the job and upload as artifacts.licenserc.toml: excludes
.clusterfuzzlitefrom the license-header check (consistent withdocker/.github).How did you test it?
cflite_prworkflow needs to exist on the default branch before it runs on PRs, so its first real exercise will be the next PR after this lands.Notes for the reviewer
If this creates more problems than it helps to solve or if it slows down the CI to much, I am happy to reconsider but I'd like us to at least try this out.
I can already tell that the fuzzer will be red when it runs for the first time. One expected finding is that fuzz_pipeline_loads fails with an AttributeError. In base.py or from_dict, we should validate that deserialized_data is a dict and raise DeserializationError otherwise. This is the kind of bug fuzzing should surface.
Checklist
ci:).🤖 Generated with Claude Code