You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implement training.disp_avg in the backend-independent training loop and enable it for pt_expt.
pt_expt currently accepts the option but reports only the latest step metrics. The legacy PT trainer has separate interval-aggregation logic, which should be replaced by one shared implementation.
Root cause
The common trainer loop has no first-class metric aggregation policy. Consequently, each backend must either implement averaging independently or silently ignore disp_avg.
Proposed design
Add a common MetricAccumulator abstraction that:
accumulates detached sums and counts per task and metric;
distinguishes training step averages from atom-weighted validation averages;
supports sparse multi-task sampling without dividing by zero;
resets only after a display interval is emitted;
optionally reduces sums and counts across ranks at display time.
Backend-neutral aggregation should live under deepmd/dpmodel/train. Torch-specific tensor/reduction helpers may live under deepmd/pt_expt/train, with legacy PT importing the same implementation.
Performance and correctness requirements
Keep accumulated values on device where possible.
Perform host synchronization only at display boundaries.
Do not fetch extra training batches merely to display inactive tasks.
Do not introduce operations into the compiled model graph.
Summary
Implement
training.disp_avgin the backend-independent training loop and enable it for pt_expt.pt_expt currently accepts the option but reports only the latest step metrics. The legacy PT trainer has separate interval-aggregation logic, which should be replaced by one shared implementation.
Root cause
The common trainer loop has no first-class metric aggregation policy. Consequently, each backend must either implement averaging independently or silently ignore
disp_avg.Proposed design
Add a common
MetricAccumulatorabstraction that:Backend-neutral aggregation should live under
deepmd/dpmodel/train. Torch-specific tensor/reduction helpers may live underdeepmd/pt_expt/train, with legacy PT importing the same implementation.Performance and correctness requirements
disp_avg=false.Acceptance criteria
disp_avg=truereports the mean over the completed display interval.disp_avg=falsereports the existing instantaneous metrics.Refs #5755.