Brazenly steal the useful parts of Ansible's Jinja standard library

[leynos]

Goal

Survey the Jinja functions, filters, and tests exposed by ansible-core 2.21.3, identify the useful capabilities that Netsuke and MiniJinja 2.12 do not already provide, and adopt them with Netsuke-native semantics.

This is an umbrella specification and prioritization issue, not a request for one colossal standard-library PR. Accepted groups should become focused implementation issues after v0.1.0 final. The current release must remain a hardening release as defined by #594.

The aim is not Ansible compatibility for its own sake. The aim is to loot a mature toolbox for operations that make declarative build manifests markedly easier to write, while refusing the Ansible-specific, nondeterministic, surprising, or security-hostile pieces.

Current Netsuke baseline

Netsuke already adds the following surface on top of MiniJinja:

• Functions: env, glob, fetch, now, timedelta, which, and command_available.
• Filters: basename, dirname, with_suffix, relative_to, realpath, expanduser, size, contents, linecount, hash, digest, uniq, flatten, group_by, shell, grep, and which.
• Tests: dir, file, symlink, pipe, block_device, char_device, and device.

MiniJinja itself already supplies many names that also appear in Ansible, including bool, default/d, groupby, map, select, selectattr, reject, rejectattr, split, unique, zip, tojson, and the ordinary Jinja type/comparison tests. Do not add Netsuke wrappers unless a concrete semantic gap justifies one.

Licensing boundary

Ansible's implementation is GPL-3.0-or-later; Netsuke is ISC licensed. We may borrow names, concepts, documented signatures, and independently verified observable behaviour, but we must not copy, transliterate, or mechanically port Ansible's Python implementation.

Implement each accepted helper independently in Rust from a written Netsuke contract and Netsuke-owned tests. Where Ansible behaviour depends on Python quirks, unstable set ordering, permissive coercion, or deprecated compatibility behaviour, deliberately specify something better.

Recommended theft, highest priority

Data interchange

Candidate	Form	Netsuke contract
from_json	filter	Parse JSON text into native MiniJinja values. Reject duplicate object keys if the selected parser can detect them, report source offsets where available, and preserve object order.
from_yaml	filter	Parse one YAML document into native values using Netsuke's existing safe YAML stack. Reject unsupported tags and duplicate keys rather than inheriting Ansible loader magic.
from_yaml_all	filter	Parse a multi-document YAML stream and return a materialized sequence. Do not expose a lazy iterator whose errors surface unpredictably later.
to_yaml	filter	Deterministically serialize native values. Define key ordering, line endings, scalar quoting, and trailing-newline behaviour.
to_nice_yaml	filter	Human-readable block-style YAML with explicit indentation options and the same determinism contract.
to_nice_json	filter	Pretty JSON with explicit indentation and optional deterministic key sorting. Keep MiniJinja's existing tojson as the compact/default JSON serializer rather than adding an unnecessary to_json alias.

These unlock direct consumption of compiler metadata, package manifests, generated configuration fragments, and bounded output from contents, fetch, or command helpers without shelling out to jq, yq, Python, or Ruby.

Mapping and sequence transforms

Candidate	Form	Netsuke contract
combine	filter	Merge mappings, with later mappings taking precedence. Support a documented recursive mode and explicit list policies. Preserve deterministic insertion order.
dict2items	filter	Convert a mapping to a sequence of {key, value} objects, with optional field names. Preserve mapping order.
items2dict	filter	Inverse of dict2items; reject missing fields and duplicate keys unless an explicit duplicate policy is supplied.
extract	filter	Resolve a key or nested key path from a mapping/sequence. Make missing-value behaviour explicit rather than silently producing undefined values.
subelements	filter	Expand parent objects against a nested child sequence, returning deterministic parent/child pairs. Support an explicit skip_missing option.
rekey_on_member	filter	Re-index a sequence or mapping of objects by one member. Default to rejecting duplicate derived keys; permit overwrite only explicitly.

combine, dict2items, and items2dict should land first. They are disproportionately useful with Netsuke's vars, foreach, per-entry overrides, and platform/toolchain configuration layering.

Set algebra and build matrices

Candidate	Form	Netsuke contract
union	filter	Ordered union, preserving first appearance across the left then right sequence.
intersect	filter	Ordered intersection in left-sequence order, with duplicates removed.
difference	filter	Ordered left difference, with duplicates removed.
symmetric_difference	filter	Ordered symmetric difference with a defined left-then-right order.
product	filter	Cartesian product for target matrices. Materialize deterministically and reject cardinality overflow before allocation.
combinations	filter	Ordered combinations, with checked output cardinality.
permutations	filter	Ordered permutations, with checked output cardinality. This should carry a lower default resource ceiling than product.
zip_longest	filter	Zip sequences to the longest input using an explicit fill value. MiniJinja already supplies ordinary zip.

Do not reproduce Ansible's set-backed unstable ordering. A Netsukefile compiled twice from the same inputs must continue to produce the same graph byte for byte.

Regular expressions

Add the Ansible-shaped filter family:

• value | regex_replace(pattern, replacement, ignorecase=false, multiline=false, count=0[, mandatory_count])
• value | regex_search(pattern, ignorecase=false, multiline=false)
• value | regex_findall(pattern, ignorecase=false, multiline=false)
• value | regex_escape([dialect])

Add the corresponding tests:

• value is match(pattern, ignorecase=false, multiline=false) for a match anchored at the start;
• value is search(pattern, ignorecase=false, multiline=false) for a substring search;
• value is regex(pattern, match_type='search', ...) for explicit search, match, or fullmatch selection.

Rust's regular-expression ecosystem does not have Python re semantics. The public contract must name the supported dialect and document unsupported features such as look-around or replacement back-reference differences. Exact compatibility is less important than a coherent, safe, well-tested Netsuke dialect. Invalid patterns must produce typed, localized template diagnostics.

Version predicates

Add a Jinja test rather than a transformation filter:

compiler_version is version('1.82.0', '>=')

Support version as the canonical name and optionally version_compare as an Ansible-discoverability alias only if aliases have an explicit compatibility policy.

Initial scope should use the existing semver dependency and support the operators ==, !=, <, <=, >, and >=, plus their mnemonic forms. Do not call permissive or ecosystem-specific parsing modes semver. PEP 440, Debian, RPM, and deliberately loose versions should become separate version schemes only when a real consumer requires them.

This is high-value for toolchain probing and conditional flag selection, and considerably clearer than ad hoc string comparison.

Recommended path and filesystem theft

Pure lexical path filters

Candidate	Purpose and required semantics
path_join	Join a sequence of path components. Define whether it follows host-platform rules or accepts an explicit path dialect.
normpath	Lexically normalize separators and ./.. without touching the filesystem. It must not escape a capability boundary merely because it normalizes text.
splitext	Return a two-element sequence containing stem and extension. Document multi-suffix behaviour rather than inheriting Python's quirks by accident.
commonpath	Return the longest common lexical path and reject incompatible roots/drives.
relpath	Compute a general lexical relative path, including ... Keep the existing stricter relative_to, which correctly rejects paths outside the supplied root.
win_basename, win_dirname, win_splitdrive	Parse Windows path strings consistently even when Netsuke runs on Unix. These are useful for cross-compilation and should be lexical, not host-observing.

Environment-backed path expansion

expandvars is useful, but unlike the lexical helpers it observes environment state. Implement it through the injected environment reader at the standard-library composition root, not direct ambient reads inside the helper. Define missing-variable behaviour, preferably strict by default with an opt-in preserve mode. Disable it during netsuke help targets queries just as env is disabled.

Filesystem tests

Add the useful Ansible file predicates that Netsuke lacks:

• path is exists: follows symlinks and reports whether the referent exists;
• path is link_exists: uses link metadata and therefore remains true for a dangling symlink;
• path is abs: pure lexical absolute-path test;
• path is same_file(other): compares file identity rather than path spelling;
• path is mount: reports mount points where the platform exposes meaningful semantics.

exists, link_exists, and abs are the priority. same_file and mount are more specialized and must remain capability-scoped and explicitly platform-qualified.

Do not add Ansible's entire alias thicket (directory, is_dir, link, is_link, is_abs, and so on) unless Netsuke adopts a general compatibility-alias policy. Existing Netsuke names are shorter and coherent.

fileglob is also unnecessary: Netsuke already has glob(). If filtering to regular files proves common, add a typed files_only=true option or compose glob() with the file test rather than introducing a second glob implementation.

Recommended collection and truth predicates

Add these tests because they compose naturally with select, selectattr, reject, and rejectattr:

• values is any
• values is all
• values is subset(other)
• values is superset(other)
• container is contains(value)
• value is truthy(convert_bool=false)
• value is falsy(convert_bool=false)

contains looks backwards in isolation, but it is valuable in expressions such as records | selectattr('tags', 'contains', 'rust') where Jinja passes the attribute value as the test subject.

For subset and superset, define duplicate handling and deterministic equality without forcing values through a hash-set representation. For truthy and falsy, retain normal MiniJinja truthiness by default. Any string-to-boolean conversion must accept a small documented vocabulary and reject unknown spellings, not copy Ansible's deprecated coercion fallback.

The nan/isnan tests are not compelling for a YAML build language and should wait for an evidenced use case.

Recommended encoding, identity, and formatting helpers

Candidate	Form	Notes
b64encode / b64decode	filters	Support text and bytes deliberately, define UTF-8 conversion, padding, and invalid-input errors.
urldecode	filter	Pair with MiniJinja's existing URL encoding support. State whether + decodes to a space.
to_uuid	filter	Deterministic UUIDv5 generation. Use a documented Netsuke namespace or require one explicitly; do not silently inherit Ansible's namespace UUID.
shell_quote	filter	Quote one value for a named shell dialect. Reuse Netsuke's existing quoting machinery where appropriate.
quote	optional alias	Add only if the alias clearly means shell quoting and cannot be confused with HTML/template escaping. Structured recipes in #593 remain the preferred shell-free solution.
comment	filter	Decorate text as comments for common generated-file syntaxes. Keep presets small and permit an explicit prefix.
human_readable	filter	Format byte/bit counts with an explicit unit system and rounding contract.
human_to_bytes	filter	Parse human-readable sizes strictly, with checked integer conversion.

Text hashing without breaking hash

Ansible's hash hashes the supplied string. Netsuke's existing hash treats the supplied string as a file path and hashes that file's contents. Overloading one name with type- or existence-dependent behaviour would be a trapdoor.

Add a distinct pure name such as text_hash or hash_text, plus an optional checksum alias if useful. Keep weak digest names (md5, sha1) behind the existing legacy-digests policy or omit them entirely. Do not silently change the current hash contract before a deliberate breaking release and migration plan.

Recommended date and time additions

Netsuke already has a stronger now(offset=...) object and timedelta(...). Steal the missing capabilities, not Ansible's exact calling convention:

• text | to_datetime(format=...) parses an explicit timestamp into Netsuke's timestamp value;
• timestamp | strftime(format) formats a timestamp;
• optionally permit now(...).format(...) if methods on the existing object provide a cleaner API.

Parsing and formatting explicit values are pure. Reading the clock remains host-observing. Locale-sensitive output must either pin a locale or require one explicitly so that identical manifests do not acquire machine-dependent graph text.

Conditional theft

Seeded randomization only

Ansible exposes random and shuffle. Unseeded variants directly violate Netsuke's deterministic-graph mandate and must not exist.

A future use case may justify deterministic forms requiring an explicit seed:

values | shuffle(seed='stable-test-partition-v1')
values | random(seed='stable-test-choice-v1')

The algorithm and seed-to-stream mapping would then become part of Netsuke's compatibility contract. Until a consumer needs this, defer both helpers.

Mathematics

Ansible's log, pow, and root filters are easy to implement but low-value for build manifests and introduce floating-point edge cases. Defer them until an actual Netsukefile requires them. Integer exponentiation could be considered separately with checked arithmetic.

Type debugging

type_debug can help diagnose template data, but MiniJinja already supplies debugging and pretty-printing facilities plus type tests. Add it only if current diagnostics do not expose the relevant value kind cleanly.

Ansible global functions: mostly do not steal

Ansible's notable global functions are lookup, query/q, now, and undef.

• now already exists in Netsuke; extend formatting as described above.
• undef conflicts with Netsuke's strict-undefined model and has little value without Ansible's variable-precedence machinery.
• A generic lookup(plugin_name, ...) or query(...) dispatcher would hide I/O, purity, capability, policy, and result cardinality behind a string. Netsuke's explicit env, glob, contents, fetch, and which helpers are easier to audit and document.

Steal useful lookup capabilities as typed named helpers. Do not add a generic plugin dispatcher unless a future extension/provider design proves that explicit functions cannot scale. Any such work should compose with the provider architecture in #590 rather than creating a second dynamic registry.

Things to leave in Ansible's shed

Do not add the following merely for name parity:

• Ansible task-result tests: failed, succeeded, changed, skipped, reachable, unreachable, timedout, started, and finished.
• Vault tests: vault_encrypted and vaulted_file.
• password_hash, which brings salt, algorithm, dependency, and secrecy concerns unrelated to build graph construction.
• mandatory, because Netsuke already treats undefined values as errors.
• ternary, because Jinja already has conditional expressions.
• bool, default, map, select, selectattr, reject, rejectattr, groupby, split, unique, zip, and to_json, where MiniJinja already supplies the capability under its normal name.
• basename, dirname, expanduser, realpath, and flatten, which Netsuke already supplies.
• Ansible's permissive boolean coercion, loose-version defaults, unstable set ordering, legacy digest defaults, and broad alias collections.

Cross-cutting implementation contract

Every accepted helper must document and test all of the following:

1. Purity class: pure, clock-observing, environment-observing, filesystem-observing, network-observing, or subprocess-observing.
2. Manifest-query availability: only pure, non-disclosing helpers are available while rendering netsuke help targets; excluded helpers fail explicitly rather than disappearing.
3. Determinism: stable output ordering, serialization, line endings, path rendering, locale, and random behaviour.
4. Capability boundary: filesystem and environment access enter through injected, testable composition-root dependencies rather than ambient reads in leaf helpers.
5. Platform contract: Unix, Windows, and unsupported-platform behaviour is explicit. Cross-platform lexical operations must not accidentally use host-native parsing when their purpose is to parse another platform's paths.
6. Type and error contract: accepted input kinds, coercions, undefined/null handling, overflow, duplicate handling, and invalid syntax are specified. Avoid Python compatibility quirks unless they are independently useful.
7. Resource bounds: parsers, combinatorial helpers, regex processing, and materialized output reject unreasonable expansion before exhausting memory.
8. Localization and diagnostics: user-facing failures use Netsuke's typed/localized diagnostic conventions.
9. Documentation: each helper receives a signature, purity label, examples, edge cases, and a runnable documentation test in docs/stdlib-yaml-and-jinja-guide.md.
10. Testing: unit tests cover semantics and errors; property tests cover round trips, ordering, merge laws, path invariants, and serialization determinism where appropriate; integration tests prove manifest expansion and the help targets restriction.

Suggested delivery slices

Create focused follow-up issues rather than one omnibus implementation:

1. Structured data: JSON/YAML parsing and serialization.
2. Mapping transforms: combine, dict2items, items2dict, extract, subelements, and rekey_on_member.
3. Collection algebra: ordered set operations, products, combinations, permutations, and zip_longest.
4. Pattern and version predicates: regex filters/tests and SemVer comparison.
5. Lexical paths and file tests: path composition/normalization plus exists, link_exists, abs, same_file, and mount.
6. Encoding and formatting: Base64, URL decoding, UUIDv5, shell quoting, comments, and human-readable sizes.
7. Date/time conversion: to_datetime and strftime over the existing timestamp object.

The first implementation wave should prioritize combine, dict2items, items2dict, ordered set operations, regex, version, path_join, normpath, splitext, exists, link_exists, from_json, and Base64. These cover the most common build-manifest contortions with relatively little conceptual sprawl.

Acceptance criteria

• The candidate matrix is reviewed and each entry is accepted, deferred, or rejected explicitly.
• Accepted capabilities are split into focused child issues with release targets after v0.1.0 final.
• Each child issue records the full cross-cutting contract above rather than saying only "match Ansible".
• Naming collisions with MiniJinja and Netsuke, especially hash, groupby/group_by, unique/uniq, quote, and now, have deliberate resolutions.
• No implementation copies or mechanically ports GPL Ansible source.
• The standard-library guide gains a maintained inventory distinguishing MiniJinja built-ins, Netsuke extensions, adopted Ansible-inspired helpers, and deliberately unsupported Ansible helpers.
• Full repository formatting, lint, type-check, test, documentation, localization, and policy gates pass for each implementation slice.

Prior art surveyed

• Ansible core filter implementation, v2.21.3
• Ansible collection and mathematical filters, v2.21.3
• Ansible URL filters, v2.21.3
• Ansible core tests, v2.21.3
• Ansible filesystem tests, v2.21.3
• Ansible collection tests, v2.21.3
• MiniJinja 2.12 filter implementation
• Netsuke's current src/stdlib implementation and docs/stdlib-yaml-and-jinja-guide.md on main.