feat(lammps): load selected dump frames efficiently

[njzjz-bot]

Summary

• add f_idx support to lammps/dump so callers can load arbitrary non-negative frame indices directly
• stream trajectory frames one at a time, retain only requested frames, and stop after the largest requested index
• preserve requested order and duplicate indices for parity with System.sub_system
• validate empty, negative, out-of-range, and conflicting begin/step selections
• split selected trajectories at their actual ITEM: TIMESTEP boundaries instead of assuming a fixed block length

Example:

system = dpdata.System(
    "trajectory.dump",
    fmt="lammps/dump",
    type_map=["O", "H"],
    f_idx=[23, 56, 78],
)

Performance benchmark

The benchmark compares direct sparse loading against the existing workflow of loading the complete trajectory and then calling sub_system(indices).

• synthetic LAMMPS dump: 12.15 MiB, 5,000 frames, 100 atoms per frame
• selection: 10 evenly distributed frames, including the final frame
• measurement: one warm-up followed by 5 measured runs per method
• each measurement ran in a fresh Python process; table values are medians
• environment: Linux 6.8.0-110-generic, Python 3.13.9, NumPy 2.5.1

Method	Median wall time	Median peak RSS
System(..., f_idx=indices)	0.092 s	39.38 MiB
System(...).sub_system(indices)	1.328 s	110.59 MiB

Direct sparse loading was 14.38x faster and reduced peak RSS by 64.39% for this workload. Both methods returned the same 10 selected frames. Because the final frame was selected, both methods traversed the full file; the improvement comes from avoiding storage and parsing of unselected frames.

Validation

• python -m unittest test_lammps_dump_skipload.py test_lammps_dump_to_system.py test_lammps_dump_unfold.py test_lammps_dump_shift_origin.py test_lammps_dump_idx.py test_lammps_read_from_trajs.py test_lammps_spin.py — 70 tests passed
• ruff check dpdata/ tests/test_lammps_dump_skipload.py — passed
• ruff format --check dpdata/ tests/test_lammps_dump_skipload.py — passed
• dpdata --help and dpdata --version — passed
• full discovery ran 2,241 tests; 12 Amber mask tests errored because the optional parmed dependency is not installed in the environment, and 43 tests were skipped

Closes #367.

Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh

Summary by CodeRabbit

• New Features

  • Added support for loading specific LAMMPS dump frames by index.
  • Preserves requested frame order and duplicate selections.
  • Supports efficient partial reading, stopping after the final requested frame.
  • Added validation for invalid frame selections and incompatible options.

• Bug Fixes

  • Improved LAMMPS trajectory frame splitting and selection reliability.

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 2 untouched benchmarks

---

_{Comparing njzjz-bot:feat/lammps-selective-frames (ce9dfa7) with master (b14e83e)}

[coderabbitai]

Warning

Review limit reached

@njzjz, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 58 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e8c034d2-01f8-4205-9b7e-e6ef5e308acb

📥 Commits

Reviewing files that changed from the base of the PR and between 376317d and ce9dfa7.

📒 Files selected for processing (2)

• dpdata/formats/lammps/dump.py
• tests/test_lammps_dump_skipload.py

📝 Walkthrough

Walkthrough

LAMMPS dump loading now supports selecting specific frame indices, including ordered duplicates, with input validation and early termination. The plugin forwards f_idx, while frame-splitting logic and tests are updated accordingly.

Changes

LAMMPS frame selection

Layer / File(s)	Summary
Frame iteration and indexed loading dpdata/formats/lammps/dump.py	Frames are parsed through a generator, indexed selections preserve order and duplicates, invalid requests raise defined exceptions, and begin/step loading uses the new iterator.
Plugin API propagation dpdata/plugins/lammps.py	from_system accepts and forwards f_idx, documenting its constraints.
Frame selection validation coverage tests/test_lammps_dump_skipload.py	Tests cover selection results, duplicate indices, early stream termination, invalid indices, bounds errors, and incompatible arguments.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant LAMMPSDumpFormat
  participant load_file
  participant DumpStream
  Caller->>LAMMPSDumpFormat: request frames with f_idx
  LAMMPSDumpFormat->>load_file: forward f_idx
  load_file->>DumpStream: iterate complete dump frames
  DumpStream-->>load_file: yield frames through last requested index
  load_file-->>LAMMPSDumpFormat: return ordered selected lines
  LAMMPSDumpFormat-->>Caller: construct selected system

Loading

Possibly related PRs

• deepmodeling/dpdata#1039: Implements matching f_idx support and tests for LAMMPS dump loading.
• deepmodeling/dpdata#1040: Contains closely related indexed loading, validation, early termination, and frame-splitting changes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: efficient selection of LAMMPS dump frames.
Linked Issues check	✅ Passed	The PR implements the requested frame-index-aware, more efficient LAMMPS dump reading and preserves requested order and duplicates.
Out of Scope Changes check	✅ Passed	The changes stay focused on LAMMPS dump frame selection, loading efficiency, and related tests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/test_lammps_dump_skipload.py (1)

87-93: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding a case for the TypeError path.

Coverage here checks empty/negative/out-of-range but not f_idx containing a non-integer element (e.g. f_idx=["a"]), which _normalize_frame_indices is supposed to reject with TypeError.

✅ Suggested addition

     def test_invalid_frame_indices(self):
         with self.assertRaisesRegex(ValueError, "must not be empty"):
             dump.load_file(self.dump_file, f_idx=[])
         with self.assertRaisesRegex(ValueError, "non-negative"):
             dump.load_file(self.dump_file, f_idx=[-1])
         with self.assertRaisesRegex(IndexError, "out of range"):
             dump.load_file(self.dump_file, f_idx=[5])
+        with self.assertRaisesRegex(TypeError, "only integers"):
+            dump.load_file(self.dump_file, f_idx=["a"])

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_lammps_dump_skipload.py` around lines 87 - 93, Add a TypeError
assertion to test_invalid_frame_indices for a non-integer frame index such as
f_idx=["a"], matching the rejection behavior of _normalize_frame_indices and
checking the expected error message.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/test_lammps_dump_skipload.py`:
- Around line 87-93: Add a TypeError assertion to test_invalid_frame_indices for
a non-integer frame index such as f_idx=["a"], matching the rejection behavior
of _normalize_frame_indices and checking the expected error message.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 04b3650b-88e9-4513-a996-777a60b2d15e

📥 Commits

Reviewing files that changed from the base of the PR and between 212ab49 and 376317d.

📒 Files selected for processing (3)

• dpdata/formats/lammps/dump.py
• dpdata/plugins/lammps.py
• tests/test_lammps_dump_skipload.py

[codecov]

Codecov Report

❌ Patch coverage is 94.59459% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.65%. Comparing base (212ab49) to head (ce9dfa7).
⚠️ Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
dpdata/formats/lammps/dump.py	94.52%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1041      +/-   ##
==========================================
+ Coverage   87.63%   87.65%   +0.02%     
==========================================
  Files          90       90              
  Lines        9209     9257      +48     
==========================================
+ Hits         8070     8114      +44     
- Misses       1139     1143       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[njzjz]

Addressed the review nitpick in b68b96d: test_invalid_frame_indices now also asserts TypeError for a non-integer element (f_idx=["a"]), for bool elements (f_idx=[True], rejected by the explicit isinstance(index, bool) guard), and for a non-iterable scalar (f_idx=1.5), so both TypeError branches of _normalize_frame_indices are covered.

[wanghan-iapcm]

The f_idx implementation itself is well done -- order/duplicate preservation, early termination, and the atom-id remap under selection all check out, and I confirmed System(f, f_idx=[1,2]) matches System(f).sub_system([1,2]) exactly. Two things I'd like addressed before merge. Note that CodeRabbit was rate limited on this PR and never actually reviewed it.

[wanghan-iapcm]

Both threads verified at ce9dfa7.

Blank-line regression: fixed and genuinely guarded. _iter_frames now skips blank separators and split_traj clamps each frame after its declared atom records. Base vs head:

input	base 212ab49	head ce9dfa7
conf.5.dump	5 frames	5 frames
+ trailing newline	5 frames	5 frames
blank line between frames	TypeError: 'NoneType' object is not subscriptable	5 frames
load_file(...) + ["# end of run"]	5 x 11 lines	5 x 11 lines

test_blank_trailer_is_ignored_and_extra_text_is_clamped fails at the pre-fix head b68b96d (AssertionError: True is not false) and passes here, so it is a real regression test.

Early-stop probe: CountingStringIO now has teeth. I re-ran the two adversarial loaders I used to break the old one -- a full-file readline() loader with no early termination, and one that stops two frames late. Both now fail (1120 not less than or equal to 463 and 911 not less than or equal to 463); the real implementation passes. The probe backs the speedup claim properly.

Also spot-checked f_idx end to end: [0], [1], [4], [3,1], [1,1,0] and the bare int 2 all match sub_system exactly, and empty, negative, out-of-range, float, string, bool and np.bool_ mask inputs all raise the documented errors. Full suite at head is 2242 tests with the same 45 pre-existing optional-dependency errors as the base run.

Thanks for the thorough hardening.