Skip to content

fix(replace): fill omitted replacement-map entries with placeholders - #287

Open
dundysm wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
dundysm:fix/278-fill-omitted-replacement-map-entries
Open

dundysm wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
dundysm:fix/278-fill-omitted-replacement-map-entries

Conversation

@dundysm

@dundysm dundysm commented Sep 23, 2026

Copy link
Copy Markdown

Summary

Fixes #278.

When LlmReplaceWorkflow.generate_map_only asks the replacement-generator LLM for a full map, entity-dense rows can occasionally drop one (value, label) pair. After #246, _prepare_rewrite_tagged_text fail-closes unless applied_span_count == targeted_span_count, so a single omission blanks the whole rewrite for that record.

This change keeps the fix inside _filter_replacement_map_to_input_entities:

  • after the existing exact-match / collision-repair pass, compute allowed_pairs - filled_pairs
  • append a deterministic placeholder via _collision_safe_synthetic(...) for each remaining pair (stable (label, original) order)
  • share the per-label index counter with collision repair so indices stay unique
  • emit one PII-free WARNING (filled_by_label=...) matching the collision-repair style
  • leave the empty-map WARNING in place (only if somehow still empty after fills)
  • keep DEBUG unfilled_by_label telemetry before the fill so LLM miss rates remain visible

Shared filter path also covers Substitute: a missing synthetic no longer leaves the original value in place.

Intentionally scoped away from draft #198 umbrella rewrite work and #179 windowed generate_map_only.

Test plan

  • uv run pytest tests/engine/test_llm_replace_workflow.py -q (13 passed)
  • uv run pytest tests/engine/test_rewrite_generation.py tests/engine/test_combined_rewrite_workflow.py -q
  • make format-check
  • new coverage: partial-map fill, index continuity with collision repair, complete map skips fill WARNING, all-omitted fill without PII in logs

cc @asteier2026 @binaryaaron @lipikaramaswamy @andreatnvidia @NVIDIA-NeMo/anonymizer-maintainers

When the replacement-generator LLM drops a requested (value, label) pair,
_filter_replacement_map_to_input_entities now appends a collision-safe
placeholder via _collision_safe_synthetic and emits a PII-free WARNING.
That keeps applied_span_count == targeted_span_count so rewrite readiness
is not lost to a single omission.

Fixes NVIDIA-NeMo#278

Signed-off-by: Dundy Pasupuleti <dundysm@gmail.com>
@dundysm
dundysm requested a review from a team as a code owner September 23, 2026 12:27
@github-actions

Copy link
Copy Markdown
Contributor

Linked Issue Check

Issue #278 has not been triaged yet. A maintainer needs to review
the issue and add the triaged label before this PR can be merged.

You can continue working on the PR in the meantime. When the issue is
triaged, the workflow will rerun this linked-issue check for open PRs
that reference the issue.

@dundysm

dundysm commented Sep 23, 2026

Copy link
Copy Markdown
Author

ready for review.

@asteier2026 @binaryaaron @lipikaramaswamy @andreatnvidia @NVIDIA-NeMo/anonymizer-maintainers

fill lives only in _filter_replacement_map_to_input_entities (reuses _collision_safe_synthetic + pii-free warning). scoped clear of draft #198 and #179 windowing.

@greptile-apps

greptile-apps Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; the prior collision issue is fully fixed and no new actionable failures were identified.

Findings

  1. P2 Placeholder values can collide ▶

Summary

This PR completes partial LLM replacement maps with deterministic, collision-safe placeholders instead of allowing omitted entities to invalidate or bypass anonymization.

  • Tracks protected originals, accepted LLM replacements, and generated placeholders as occupied values.
  • Shares placeholder counters across labels that normalize to the same token.
  • Adds coverage for partial and fully omitted maps, counter continuity, token collisions, accepted-synthetic collisions, warnings, and PII-safe logging.
  • The previous placeholder-collision finding is fully addressed.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Requested entity pairs] --> B[Filter exact LLM map entries]
    B --> C[Track originals and accepted synthetics]
    C --> D[Repair synthetic-original collisions]
    D --> E[Find omitted entity pairs]
    E --> F[Mint collision-safe placeholders]
    F --> G[Completed replacement map]
Loading

Reviews (2) · Last reviewed commit: "fix(replace): harden placeholder collisi..."

Comment on lines +233 to +239
"synthetic": _collision_safe_synthetic(
label,
index=synthetic_collision_labels[label],
protected_original_values=protected_original_values,
),
}
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Placeholder values can collide

Labels such as foo-bar and foo_bar normalize to the same placeholder token, but their counters remain separate, so omitted entries can receive the same synthetic value. A generated placeholder can also match an accepted LLM synthetic because the helper checks only original values. This violates the workflow's requirement that every entity have a different replacement and can reduce output quality. Please track all occupied synthetic values when generating placeholders and cover both collision cases with tests.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hardened in 9537e86: occupied_values now seeds from protected originals and grows with every accepted llm synthetic and minted placeholder; _collision_safe_synthetic skips any occupied value (not only originals). placeholder index counters are keyed by normalized label token so foo-bar / foo_bar share one FOO_BAR sequence. added unit tests for the cross-label token clash and for an accepted llm synthetic equal to the would-be placeholder.

Track occupied values (protected originals plus accepted LLM synthetics
and minted placeholders) when minting [SUBSTITUTE_*] fills. Key the
placeholder index counter by normalized label token so labels that
collapse to the same token cannot emit duplicate synthetics.

Signed-off-by: Dundy Pasupuleti <dundysm@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replacement-map generation can silently omit a requested entity, causing the rewrite to be marked unavailable

1 participant