BenchFlow is a universal environment framework: it runs AI agents against task environments and scores them through one hardened contract. A benchmark is just a frozen environment — point BenchFlow at any of them, drive it with any ACP agent, and run single-agent, multi-agent, or multi-round patterns over the same Scene-based lifecycle.
For local Claude Code or Codex jsonl formatted trajectory files that you are proud of (contain challenging tasks you dealed with via chatting to agents), you can upload that file to BenchFlow to join the competition for winning $2,000 cash reward. No BenchFlow account, credentials, or API key is required:
# Install or upgrade BenchFlow
uv tool install --python 3.12 --upgrade benchflow
# Upload one capture
bench traj upload /absolute/path/to/trial \
--github-id YOUR_GITHUB_ID \
--email YOU@example.comThe path may be a trial directory, a directory of JSONL files, or one JSONL
file. Both contributor fields are required and are stored in manifest.json.
See the concise upload skill
or the trajectory upload guide.
No OpenAI or Anthropic API key is required. Start Docker, install BenchFlow, then run one of these options. BenchFlow detects the saved host login and makes it available to the agent inside the sandbox.
uv tool install --python 3.12 --upgrade benchflow
docker info >/dev/null # Docker must be runningInstall the Codex CLI, then:
codex login
unset OPENAI_API_KEY CODEX_API_KEY # ensure subscription auth is used
bench eval run \
--source-repo benchflow-ai/skillsbench \
--source-path tasks/citation-check \
--agent codex \
--model gpt-5.5 \
--sandbox dockerInstall Claude Code, then:
claude auth login
unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN # ensure subscription auth is used
bench eval run \
--source-repo benchflow-ai/skillsbench \
--source-path tasks/citation-check \
--agent claude \
--model claude-sonnet-4-6 \
--sandbox dockerThe agent may pass or fail the benchmark task; either result means the
evaluation completed. Each run writes rewards, token usage, and the full
trajectory under jobs/. See Getting started for
other agents, models, and sandboxes.
Install or upgrade to the latest stable release from PyPI with uv:
uv tool install --python 3.12 --upgrade benchflow- Confirm with
bench --version. - BenchFlow CLI releases require Python 3.12 or newer. Keep
--python 3.12in the install command souvdoes not resolve an older Python-compatible package that lacks the CLI entrypoints. - If you see
Executables already exist: bench, benchflow, re-run withuv tool install --python 3.12 --upgrade --force benchflowto replace stale entrypoints from an older install. - For Daytona, Modal, or AgentCore extras, install the relevant optional package, for example
uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'.
Internal users wanting the newest preview from main install the internal preview channel (uv tool install --python 3.12 --prerelease allow --upgrade benchflow).
Requirements & auth. Install uv; the
--python 3.12 flag lets it provision a compatible interpreter for the tool
install. Set DAYTONA_API_KEY for Daytona or configure Modal auth for Modal;
export an agent API key (GEMINI_API_KEY, ANTHROPIC_API_KEY, …) or use
subscription auth (claude auth login / codex login). Provider-prefixed models
may need provider-specific credentials; Azure Foundry uses AZURE_API_KEY +
AZURE_API_ENDPOINT.
Start with Getting started, then Concepts for the mental model. Prefer to have an AI coding agent run the whole quickstart for you? Paste the agent quickstart prompt into Claude Code, Codex CLI, or Gemini CLI. Then by goal:
| If you want to… | Read |
|---|---|
| Run an eval on an existing task | Getting started |
| Understand how BenchFlow runs any benchmark (the three-layer model) | Run any benchmark |
| Have an AI agent install + run the quickstart end to end | Agent quickstart prompt |
| Run an agent from the public agents repo (goose, qwen-code, prime-agent, …) | Running external agents |
| Understand Rollout / Scene / Role / Verifier | Concepts |
| Author a new task | Task authoring |
Author a task in the native task.md format |
Native task.md authoring |
| Run a hosted PrimeIntellect / Verifiers environment | CLI reference |
| Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | Use cases |
| Multi-round single-agent (progressive disclosure, oracle access) | Progressive disclosure |
| Skill evaluation (when the artifact is a skill, not a workspace) | Skill eval |
| Contribute a trajectory capture | Trajectory upload |
| Understand the security model | Sandbox hardening |
| Use public vs internal preview SDK releases | Release channels |
| CLI flags + commands | CLI reference |
| Python API surface | Python API reference |
Notebooks and runnable example scripts live under docs/examples/ so examples stay versioned with the docs that explain them.
bench agentvsbench eval adopt.bench agent list/bench agent showinspect registered AI agents (the solver programs like Claude Code or Gemini CLI). Onboarding a third-party benchmark intobenchmarks/<name>/is a separate workflow —bench eval adopt <source>scaffolds and drives the conversion, andbench eval adopt <name> --verifyparity-gates it. (The legacybench agent create|run|verifycommands still work as deprecated aliases.) See the CLI reference for details.
Benchmark datasets live in external Git repos and are referenced with two fields:
# benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml
source:
repo: benchflow-ai/benchmarks # GitHub org/repo
path: datasets/harvey-lab/tasks # optional subpath within repo
ref: main # optional branch/tag
agent: gemini
model: gemini/gemini-3.1-flash-lite-previewRun any benchmark via the CLI:
# From a YAML config (shipped with the repo)
bench eval run --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml
# Inline — mirrors the YAML source fields
bench eval run \
--source-repo benchflow-ai/skillsbench --source-path tasks \
--agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64Repos are cloned and cached locally under .cache/datasets/ on first use.
Hosted environments are another source type. Instead of a repo, pass
--source-env with the environment's pinned source version to run an external
PrimeIntellect / Verifiers environment on its own native harness — BenchFlow
preserves the hosted identity (env_uid, hub_url) and still writes the shared
rollout output contract. See the CLI reference for
the full hosted-environment command shape.
Downstream projects should depend on the public PyPI release by default. For internal validation before the next public release, install or lock the internal preview channel with prereleases enabled; see Release channels.
A task is one task.md (YAML frontmatter for config + a markdown prompt body)
plus environment/ and verifier/ sidecars. The bench tasks commands cover
the authoring lifecycle:
bench tasks init my-task # scaffold a task.md package under tasks/
bench tasks check tasks/my-task # validate (default --level structural)
bench tasks migrate legacy-task/ --remove-legacy # convert old split packages to task.md
bench tasks export tasks/my-task out/ # write a compatibility export + loss reportSee Native task.md authoring and the task standard.
- Progressive disclosure on SWE-bench Pro — the
BaseUserabstraction drives a multi-round rollout: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo atdocs/examples/swebench_pro_progressive_disclosure.ipynb. See Progressive disclosure.
- Eval researchers / paper writers → Getting started → Concepts → Use cases
- Task authors → Task authoring → Sandbox hardening
- Agent builders integrating with benchflow → Concepts → Python API reference →
benchflow.agents.registry - External benchmark adapters → Task authoring → Progressive disclosure
PRs welcome. Open against main. CI runs ruff + tests on every PR; please run ruff check . and pytest tests/ locally first.
Release channels are documented in Release channels. In
short: merges to main publish an internal preview after CI passes, while a
matching release tag publishes the public release.
Apache-2.0.