Skip to content

Add training-fitted normalization to time-series processors - #1231

Open
LIU-kaiyu wants to merge 1 commit into
sunlabuiuc:masterfrom
LIU-kaiyu:liu-kaiyu/491-timeseries-normalization
Open

Add training-fitted normalization to time-series processors#1231
LIU-kaiyu wants to merge 1 commit into
sunlabuiuc:masterfrom
LIU-kaiyu:liu-kaiyu/491-timeseries-normalization

Conversation

@LIU-kaiyu

Copy link
Copy Markdown

Fixes #491.

Adds optional per-feature z-score normalization to TimeseriesProcessor and TemporalTimeseriesProcessor through normalize_strategy="standard". Normalization remains disabled by default.

The processors incrementally fit the training mean and population standard deviation after resampling and imputation, then apply those fixed statistics to subsequent samples. Each resulting timestep contributes equally; constant features use a scale of 1. Temporal timestamps remain unchanged.

The change also:

  • Validates normalization inputs and requires successful fitting before processing.
  • Clears previous statistics when refitting, including after a failed refit.
  • Preserves fitted statistics through existing serialization and cache fingerprints.
  • Adds documentation and a synthetic example showing patient-level splitting before fitting and explicit reuse of training processors for validation/test data.

Validation: 39 focused tests passed in Pixi, including 20 normalization tests covering defaults, imputation, invalid inputs, refitting, serialization, cache fingerprints, and processor transfer. The synthetic example and local contribution checks passed.

A broader run encountered three dataset-backed schema-test failures during synthetic MIMIC loading because of Windows URL/path handling; those are outside this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add normalization support in pyhealth/processors/timeseries_processor.py

1 participant