Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
841fa3a
wip: gpu-direct
jychoi-hpc Sep 3, 2026
e588da7
add
jychoi-hpc Sep 3, 2026
31f5269
fix for unittest
Sep 3, 2026
ee94573
gpudirect for vae
jychoi-hpc Sep 3, 2026
8c56872
wip: gpudirect vae
jychoi-hpc Sep 3, 2026
0816225
wip: gpudirect vae
jychoi-hpc Sep 3, 2026
f03ab78
add DDSTORE_GPU_SYNC
jychoi-hpc Sep 3, 2026
15152de
profile
jychoi-hpc Sep 3, 2026
6212df1
update README
jychoi-hpc Sep 3, 2026
bdc58f3
fix performance issue
Sep 3, 2026
dc2abd2
logo
jychoi-hpc Sep 4, 2026
5d60ae7
logo
jychoi-hpc Sep 4, 2026
0aed9bc
logo
jychoi-hpc Sep 4, 2026
a8e6a24
add
jychoi-hpc Sep 11, 2026
5060d86
release GIL
jychoi-hpc Oct 3, 2026
7200558
add ThreadDataLoader
jychoi-hpc Oct 3, 2026
9baa4b5
add
jychoi-hpc Oct 3, 2026
63b6242
Confirm ThreadDataLoader lock necessity, find and guard GPU pool race
jychoi-hpc Oct 4, 2026
20f02f3
base
jychoi-hpc Oct 4, 2026
8b88a53
Simplify add()/get(), drop mpi4py.rc settings, document threading fin…
jychoi-hpc Oct 4, 2026
987585a
README: drop mpi4py.rc mentions from MPI thread level note
jychoi-hpc Oct 4, 2026
c822230
ThreadDataLoader: collate in the worker thread
jychoi-hpc Oct 4, 2026
80c3c65
Fix stale comments/README; free() releases owned buffers; init join()…
jychoi-hpc Oct 4, 2026
533666c
job-vae-core-extra.sh: fix cross-step RDMA on Frontier, add --num-wor…
jychoi-hpc Oct 4, 2026
3dadb32
Add get() profiling, bench_get.py, and VAE --replicate/--image-scale
jychoi-hpc Oct 4, 2026
ddcadd3
Add plan for batched get (get_batch)
jychoi-hpc Oct 4, 2026
cb1761b
Add batched get: get_batch() and DistDataset.__getitems__
jychoi-hpc Oct 4, 2026
ca77463
README: state that batched reads are the default
jychoi-hpc Oct 4, 2026
46e006e
Method 0 get_batch: MDLoader-style collective (Allgatherv + Alltoallv)
jychoi-hpc Oct 4, 2026
3e5af49
Method 0 get_batch: bound each Alltoallv round (DDSTORE_ALLTOALL_MAX_…
jychoi-hpc Oct 4, 2026
8c03569
Tests follow DDSTORE_FABRIC; add Perlmutter check script and checklist
jychoi-hpc Oct 4, 2026
eec4faf
Perlmutter results: colocate needs --overlap; platform notes
jychoi-hpc Oct 4, 2026
7f37bcc
job-vae-core-extra.sh: default to --layout=split-node
jychoi-hpc Oct 4, 2026
f386690
README: usage-focused rewrite; move measurements to docs/results.md; …
jychoi-hpc Oct 4, 2026
fffaedd
README: drop the sentence introducing the MDLoader citation
jychoi-hpc Oct 4, 2026
c12c5db
Set version to 3.0
jychoi-hpc Oct 4, 2026
2d5017e
VAE job scripts: run on both Frontier and Perlmutter
jychoi-hpc Oct 5, 2026
4791027
ThreadDataLoader: fix clean() hang; match DataLoader RNG and batch_si…
jychoi-hpc Oct 5, 2026
407935b
perlmutter-check.sh: DDP steps see all 4 GPUs (--gpus-per-node=4)
jychoi-hpc Oct 5, 2026
65049f3
Remove perlmutter-check.sh and checklist; keep findings in docs/resul…
jychoi-hpc Oct 5, 2026
36a2b9b
Format Python with black (target py311)
jychoi-hpc Oct 5, 2026
39fadd3
Remove docs/plan-batched-get.md (implemented)
jychoi-hpc Oct 5, 2026
5bf2e2a
Set version to 2.0
jychoi-hpc Oct 5, 2026
e3bd2f0
Add pyddstore.torch: general DistDataset, DistDatasetReader, ThreadDa…
jychoi-hpc Oct 5, 2026
71c94ae
DistDataset: chunk_size for chunked loading
jychoi-hpc Oct 5, 2026
4b65e82
DistDataset: accept numpy structured records as items or fields
jychoi-hpc Oct 5, 2026
dfff88a
README: 2.0 overview, PyTorch extra, library env vars, test_torch
jychoi-hpc Oct 5, 2026
0e6e6e3
DistDatasetReader: clear error when n_core is missing
jychoi-hpc Oct 5, 2026
96ad8c2
test_gpu_rdma: barrier before free() in matrix and sync_before_get
jychoi-hpc Oct 5, 2026
322e2f1
Large rows and reusable receive buffers for method 1/2 reads
jychoi-hpc Oct 5, 2026
f823257
ThreadDataLoader: separate iterator; setup.py: regenerate on NumPy ch…
jychoi-hpc Oct 5, 2026
5439e07
ThreadDataLoader: submit the next batch as soon as one is taken
jychoi-hpc Oct 5, 2026
688124e
pyddstore.torch: read_rows, alloc/out=, WindowedDataset, row_of
jychoi-hpc Oct 5, 2026
50d31d3
ThreadDataLoader: reuse_buffers, a loader-owned pool of registered bu…
jychoi-hpc Oct 5, 2026
3699e57
DistDataset: encode/decode hooks and fields= selection
jychoi-hpc Oct 5, 2026
954ef59
Documentation site (Sphinx + MyST) for GitHub Pages; shorter README
jychoi-hpc Oct 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: docs

# Build the Sphinx docs on every push and pull request; publish them to
# GitHub Pages from main. Needs Settings > Pages > Source: "GitHub Actions".
on:
push:
pull_request:
workflow_dispatch:

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: pages-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r docs/requirements.txt
- run: sphinx-build -W --keep-going -b html docs docs/_build/html
- uses: actions/upload-pages-artifact@v3
with:
path: docs/_build/html

deploy:
if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
# Cython build output
build/
src/pyddstore.cpp
src/pyddstore/_core.cpp
src/pyddstore/_core.numpy-version
# built module
PyDDStore.egg-info/
pyddstore.cpython-*.so
_core.cpython-*.so
__pycache__/
*.pyc
# Sphinx output
docs/_build/
8 changes: 4 additions & 4 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,14 @@ target_include_directories(demo PUBLIC ddstore)
# option(BUILD_PYTHON_BINDINGS "Build Python bindings using cython" ON)
# find_package(Python3 REQUIRED COMPONENTS Interpreter Development NumPy)
# add_custom_command(
# OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/src/pyddstore.cpp
# COMMAND ${Python3_EXECUTABLE} -m cython -3 --cplus ${CMAKE_CURRENT_SOURCE_DIR}/src/pyddstore.pyx
# OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/src/pyddstore/_core.cpp
# COMMAND ${Python3_EXECUTABLE} -m cython -3 --cplus ${CMAKE_CURRENT_SOURCE_DIR}/src/pyddstore/_core.pyx
# WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/src
# DEPENDS
# ${CMAKE_CURRENT_SOURCE_DIR}/src/pyddstore.pyx
# ${CMAKE_CURRENT_SOURCE_DIR}/src/pyddstore/_core.pyx
# )
# add_library(pyddstore SHARED
# ${CMAKE_CURRENT_SOURCE_DIR}/src/pyddstore.cpp
# ${CMAKE_CURRENT_SOURCE_DIR}/src/pyddstore/_core.cpp
# )
# set_target_properties(pyddstore PROPERTIES PREFIX "")
# set_target_properties(pyddstore PROPERTIES POSITION_INDEPENDENT_CODE ON)
Expand Down
Loading
Loading