Add dispatch: cron orchestration for the ingest pipeline - #9
Open
CodyCBakerPhD wants to merge 14 commits into
Open
Add dispatch: cron orchestration for the ingest pipeline#9CodyCBakerPhD wants to merge 14 commits into
CodyCBakerPhD wants to merge 14 commits into
Conversation
…ngest
Adds dispatch/dispatch.py, a generic entrypoint intended to be run on a
schedule by the self-hosted runner in data-ingest-runner. Per registered
project (dispatch/projects.yaml) it:
1. dandi-downloads the incoming dandiset from dandi.emberarchive.org
2. diffs discovered sessions against a per-project manifest
(<standardized_dir>/.ingest_state.json) to find unconverted sessions,
also forcing a full reprocess if the conversion script's own contents
(sha256) have changed since the manifest was last written
3. runs the lab's conversion command over new/changed sessions
4. dandi-uploads the standardized output
registry.py validates projects.yaml; state.py owns the manifest read/write.
Includes unit tests (registry validation, manifest round-trip, command
templating/dry-run with subprocess mocked out) wired into CI as a new
Dispatch job in test.yml. Kemere is registered as the first (and so far
only) project, with placeholder dandiset ids to be filled in once assigned.
CodyCBakerPhD
marked this pull request as ready for review
August 6, 2026 20:14
projects.yaml -> projects.json (stdlib json, drops the pyyaml dependency). Session discovery pulled out of the project registry into a new dispatch/sessions.json + sessions.py: a single glob per project didn't generalize (labs may need multiple raw subtrees or exclusions), and that shape can evolve independently of a project's dandiset ids/conversion command. sessions.json is keyed by lab, each entry an include-glob list (unioned) plus an optional exclude-glob list (matched by basename or path relative to the incoming project dir). dispatch.py now loads both registries and looks up each project's session spec by lab name (erroring per-project, not fatally, if one is missing). Tests updated/added for both files; all 21 pass, ruff clean.
CodyCBakerPhD
commented
Aug 6, 2026
Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>
Registers the ai_generated pytest marker repo-wide in the root pyproject.toml and applies it (via module-level pytestmark) to every test in dispatch/tests/, since that whole suite was AI-authored this session. Adds CLAUDE.md documenting the rule for future AI-agent sessions: mark AI-generated tests with this marker (module-level pytestmark by default, per-function @pytest.mark.ai_generated when a module mixes AI-generated and human-written tests), and register new suites against the same root pytest config rather than adding a competing one.
setuptools needed an explicit (empty) [tool.setuptools] py-modules = [] block to build a code-less, dependency-only package; hatchling doesn't need a placeholder section for that -- 'bypass-selection = true' under [tool.hatch.build.targets.wheel] says outright that the wheel ships no files, which is what py-modules = [] was standing in for anyway.
Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>
…gistry paths
Adds dispatch/schema/{projects,sessions}.schema.json, referenced from
both registry files via a $schema pointer so editors with JSON Schema
support give inline validation/autocomplete (required fields, the
six-digit dandiset id pattern, unknown-field typos). A new
dispatch/tests/test_schema.py validates the committed registries against
these schemas in CI and exercises each schema against a few known-bad
shapes, using jsonschema (new test-extra dependency).
dispatch.py's existing --registry/--sessions flags already accept any
path, not just the committed files -- documented this in dispatch/README
as the way to host the registry outside this repo (e.g. on the
self-hosted runner host) so it can be edited without a commit/PR; see
the paired data-ingest-runner change wiring REGISTRY_PATH/SESSIONS_PATH
through to those flags.
… runner host
Adds an optional container_image field to a project entry: when set,
dispatch.py's convert step runs 'docker pull' + wraps convert_command in
'docker run --rm' (bind-mounting repo_root/incoming_dir/standardized_dir
at identical host paths, so the templated {repo_root}/{incoming_dir}/
{standardized_dir} tokens need no rewriting) instead of assuming the
runner host has the lab's runtime dependencies (e.g. FFmpeg) installed
directly. Matches each lab's own published image (see labs/<lab>/
containers/, built by build_and_upload_docker_image.yml) -- the image
holds only the environment, code and data are supplied at run time, same
as the Kemere README's own documented 'docker run -v ...' usage.
Registers container_image in registry.py/the JSON Schema, sets it for
Kemere (ghcr.io/brain-bbqs/kemere-r34da059514-ingest:latest), and adds
unit tests for containerize() and the pull+run flow. All 30 tests pass,
ruff clean.
…in convert_command
Adds an optional metadata object to a project entry: string values (e.g.
{"species": "Ovis aries"}) available as convert_command placeholders
alongside {repo_root}/{incoming_dir}/{standardized_dir}, so a value like
this lives in the registry entry rather than baked into the command
tokens. Keys may not reuse the three reserved placeholder names (enforced
in both registry.py and the JSON Schema).
Kemere's --species Ovis aries is now {species} + metadata.species. Also
sets Kemere's standardized_dandiset_id to the real assigned id (000525) --
incoming_dandiset_id (000477) is still a placeholder.
Added tests for metadata read/default/shadow-rejection in both registry.py
and its schema. All 35 tests pass, ruff clean, pre-commit clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbcD48t2AYkB1YdLA2zQCm
…r-required --incoming-root/--standardized-root are now optional, defaulting to ember-incoming/ember-standardized siblings of --repo-root (created as needed) -- dispatch.py picks the location itself rather than requiring the caller to supply a correct one every time. Whatever ends up in --repo-root/--incoming-root/--standardized-root (explicit or defaulted) is now resolved to an absolute path before use. This fixes a real bug, not just a style nit: a relative value reaches 'docker run -v <host>:<container>' as a relative host path, which Docker rejects outright -- a real (non-dry-run) conversion would have failed even though the dry-run log looked fine, since dry-run only prints the command instead of running it. Added tests for both the default-path computation and the relative -> absolute resolution (via main(), with process_project mocked out). All 37 tests pass, ruff/pre-commit clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbcD48t2AYkB1YdLA2zQCm
… containers
convert_command no longer needs to name a metadata key: every entry in
metadata is now automatically appended as its own --<key> <value> flag
(underscores -> dashes), on top of the {key} templating that already
existed. Kemere's convert_command drops '--species {species}' entirely --
just 'metadata: {species: Ovis aries}' now produces '--species Ovis
aries' on the argv without the command template needing to know that key
exists.
Also forwards DANDI_API_KEY into a project's container_image when set in
dispatch's own environment (docker run -e DANDI_API_KEY, by name only --
never as a literal value on the argv, so it doesn't leak through ps/logs),
so a lab's conversion step can authenticate to DANDI too, not just the
dandi download/upload steps dispatch itself runs on the host (which
already inherit any env var set on dispatch's own process).
Tests added for both. All 39 tests pass, ruff/pre-commit clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbcD48t2AYkB1YdLA2zQCm
Flag that whether a plain DANDI_API_KEY env var authenticates a *named* dandi-cli instance (-i emberarchive) rather than only the default one depends on the installed dandi-cli version -- verify it, falling back to 'dandi login -i emberarchive' if the env var isn't picked up per-instance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbcD48t2AYkB1YdLA2zQCm
…ner host Adds dispatch/containers/dandi.Dockerfile -- a minimal python:3.13-slim + 'pip install dandi' image, published the same way as a lab's own container_image (build_and_upload_docker_image.yml). A new --dandi-image flag (default: ghcr.io/brain-bbqs/dandi-cli:latest) now wraps both dandi_download and dandi_upload in 'docker pull' + 'docker run', instead of shelling out to a 'dandi' binary the runner host must have installed directly -- the actual bug behind the first non-dry-run failure (FileNotFoundError: 'dandi'). Refactored the docker-wrapping logic shared by all three containerized steps (dandi download, dandi upload, lab conversion) into a single docker_run_prefix() helper (mounts + workdir + env forwarding), used by containerize() and the two dandi functions alike. DANDI_API_KEY forwards into every container the same way (by name only). With this, the runner host itself only needs python3 (for dispatch.py's own orchestration) and docker -- no dandi CLI install required anymore. Tests added for both containerized dandi functions and the shared prefix helper. All 41 tests pass, ruff/pre-commit clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbcD48t2AYkB1YdLA2zQCm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces
dispatch, a cron-driven orchestration system that automates the data ingest pipeline for registered labs. It coordinates downloading raw data from DANDI, detecting new sessions, running lab-specific conversion scripts, and uploading standardized output.Key Changes
dispatch.py: Main orchestration script that:
dandi downloaddandi upload--dry-run,--skip-download,--skip-upload, and--only <lab>flagsregistry.py: Project registry loader that:
projects.yamlto define lab configurationsProjectdataclass with script path resolutionstate.py: Per-project manifest manager that:
.ingest_state.jsonprojects.yaml: Registry configuration with Kemere lab as initial example, defining:
Comprehensive test suite (
dispatch/tests/):Documentation (README.md): Usage guide, layout explanation, credential setup, and project onboarding instructions
Environment declaration (
dispatch/envs/pyproject.toml): Minimal dependencies (pyyaml, pytest for tests)CI integration: Added
Dispatchjob to.github/workflows/test.ymlto run unit tests on every commitNotable Implementation Details
overwrite_flagto handle breaking changes{repo_root},{incoming_dir},{standardized_dir}substitutionhttps://claude.ai/code/session_01NbcD48t2AYkB1YdLA2zQCm