Skip to content

feat(interface): accept in-memory text records #281

Description

@afourniernv

Priority

Medium

Problem

Applications that already hold text in memory must write temporary CSV or Parquet files before calling Anonymizer. That adds avoidable I/O and complicates service integrations. Callers also need a stable way to correlate completed rows with their own request or event IDs, including when an input column name collides with a generated output column.

Proposed solution

  • Add typed TextRecord(id, text) and TextRecordsInput input shapes.
  • Let the Python API accept TextRecordsInput alongside the existing file-backed AnonymizerInput without changing AnonymizerInput.source or its JSON schema.
  • Preserve caller IDs and input order in the public result dataframe.
  • Keep the CLI file-only; do not expose nested record objects as command-line flags.
  • Retain existing file input behavior and validate configured ID columns.

Alternatives considered

  • Temporary files: workable, but expensive and awkward for request-oriented services.
  • Accepting a raw dataframe: broader and less explicit than the two fields the pipeline needs.
  • Reading private trace_dataframe state: not an appropriate public correlation contract.

Compatibility

Existing AnonymizerInput(source=...) calls, generated schemas, and CLI behavior remain unchanged. The new shape is additive.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions