Skip to content

feat(deps): metadata-driven require(extra) dependency guard#339

Merged
voorhs merged 16 commits into
devfrom
fix/322-require-extra-resolver
Jun 23, 2026
Merged

feat(deps): metadata-driven require(extra) dependency guard#339
voorhs merged 16 commits into
devfrom
fix/322-require-extra-resolver

Conversation

@voorhs

@voorhs voorhs commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replaces the import-based require(dependency, extra=None) guard with a metadata-driven require(extra: str) that reads the installed distribution metadata (importlib.metadata) and verifies every dependency of an autointent extra — recursively, including nested third-party extras like transformers[torch] → accelerate — is both installed and version-satisfied, raising a single aggregated, actionable ImportError otherwise.

Closes #322.

Why

The old guard had two structural weaknesses:

  1. Manual sync burden — each call site hard-coded a module name + extra label that had to be kept in sync with pyproject.toml by hand.
  2. No version checking — it only checked importability, never whether the installed version satisfied the declared constraint.

#322 is the concrete failure: BertScorer called require("transformers", "transformers"), which passed whenever transformers imported — even when accelerate (pulled by transformers[torch], needed by HF Trainer) was absent. Users hit a raw, deep ImportError from inside Trainer instead of our "install autointent[transformers]" message. This is now fixed structurally, with no hand-added accelerate guard — the recursion into transformers' own torch extra reaches accelerate automatically.

What changed

  • New module src/autointent/_deps.py — resolves an extra's full recursive leaf-requirement set via importlib.metadata + packaging, cycle-guarded and memoized, and validates each leaf. pyproject.toml is not shipped in wheels, but the build bakes the same requirements into the installed metadata, which is read at runtime.
  • Migrated all ~23 call sites across 10 modules to the single-extra-name form (require("transformers"), require("peft"), require("sentence-transformers"), …); removed the old require + its now-unused import importlib from _utils.py.
  • packaging promoted to a core dependency (packaging (>=23.2), no upper cap) — it was previously only transitive, which is the same latent-reliance mistake behind bert scorer: opaque ImportError when accelerate is missing (require() guard checks only transformers) #322.
  • Updated a stale prose reference in user_guides/advanced/02_embedder_configuration.py.

Error shape (aggregated, versioned)

ImportError: Feature requires extra 'transformers', but dependencies are missing or outdated:
  - accelerate>=0.26.0 (not installed)
  - transformers>=4.49.0,<5.0.0 (installed: 4.30.0)
Install with: pip install 'autointent[transformers]'

Testing

tests/test_deps.py (13 tests) injects a synthetic requirement graph via monkeypatch so assertions don't depend on what CI installs: all-present (no raise), missing leaf, outdated version, nested-extra accelerate missing (the #322 regression), cyclic extras terminate, unknown-extra ValueError. Plus one real-metadata smoke test for wiring.


🤖 Generated with Claude Code

voorhs and others added 16 commits June 22, 2026 23:10
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…etadata directly (#322)

CI runs 'mypy src/autointent tests' under strict on py3.10; the test module
needed full annotations and patching via the shared importlib.metadata module
object (deps.metadata tripped no-implicit-reexport attr-defined).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@voorhs voorhs changed the title feat(deps): metadata-driven require(extra) dependency guard (#322) feat(deps): metadata-driven require(extra) dependency guard Jun 23, 2026
@voorhs voorhs marked this pull request as ready for review June 23, 2026 20:37
@voorhs voorhs merged commit 483ead2 into dev Jun 23, 2026
19 checks passed
@voorhs voorhs deleted the fix/322-require-extra-resolver branch June 23, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bert scorer: opaque ImportError when accelerate is missing (require() guard checks only transformers)

1 participant