Add Docker workflows for GPU training, serving and offline deployment - #25
Open
KraHsu wants to merge 7 commits into
Open
Add Docker workflows for GPU training, serving and offline deployment#25KraHsu wants to merge 7 commits into
KraHsu wants to merge 7 commits into
Conversation
Provide a locked CUDA 12.8 image, runtime UID mapping, persistent caches and outputs, GPU smoke checks, and serving/training/development/worktree configurations. Preserve training signal delivery and configured inference endpoints.
Resolve Make image selection through the effective Compose configuration and export versioned bundles with image-bound configuration and integrity checks. Document complete online/offline workflows and cover image overrides, mounts, worktrees, runtime identities and bundle delivery.
Build and load the linux/amd64 image in GitHub Actions, validate Compose profiles, run container CPU checks and exercise Docker integration workflows. Include Docker tooling in the standard lint job.
Record the offline H100 inference and training/resume acceptance, subsequent runtime and Compose checks, and the validation limits for each tested image revision.
Docker inspect defaults an untagged repository to latest, while Docker save archives every tag in that repository. Resolve the default tag before saving and record the same explicit reference in the bundle manifest and environment. Preserve explicit tags, digests and image ID inputs, including registry ports. Extend the real offline round-trip test with two distinct images in one local repository and inspect the saved tar to ensure only the selected tag is present. Cover reference forms in the CLI tests and document the default tag behavior. Validation: the new integration regression fails before the fix (2 images instead of 1). The rebuilt image passes make docker-check and make docker-integration-check: 1970 CPU tests and 9 Docker integration tests.
Bind digest and image-ID exports to a content-derived local tag, preserve the source reference, and refuse conflicting delivery tags. Verify a real digest pull, export, offline load, and Compose creation using a temporary loopback registry; cover stable tags and collisions in CLI tests.
Put a private per-container compatibility directory on the image's library search path. The non-root bootstrap atomically selects the opt-in CUDA libraries there, so the entrypoint, docker exec, and restarted processes use the same loader configuration. Leave the directory inactive by default and report missing compatibility libraries before starting the requested command. Document recreation after changing the setting. The real regression checks enabled and disabled modes, arbitrary UID/GID, exec, restart, and missing libraries; it fails against the previous image at the exec comparison. Validation: the rebuilt CUDA image passes make docker-check and make docker-integration-check (1972 CPU tests and 11 integration tests), including the separate digest bundle fix and offline Compose creation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenWAM currently requires a manually prepared Python/CUDA environment. This adds a pinned CUDA 12.8, Python 3.12 and PyTorch 2.7.1 image for serving, training and source development, plus offline delivery to GPU hosts without internet access.
Changes
.env, environment variables and Compose overrides.docker execand after container restarts.:latest; digest and image-ID selections receive content-derived delivery tags that survive offline loading. Weights and datasets are transferred separately.Validation
openwam:digest-compat-fixes.make docker-check: dependency, serve CLI and Ruff checks passed; 1,972 CPU tests passed, 12 skipped and 21 deselected.make docker-integration-check PYTHON=python3: 11 tests passed, including digest export/load after shutting down the source registry, Compose creation from the delivered bundle, worktree Git operations, arbitrary runtime IDs and CUDA library selection acrossrun,execand restart.Earlier H100 acceptance covers released-checkpoint inference and four-GPU training/recovery through steps 1–20. That GPU run predates the final runtime/export fixes; current-head GPU training, multi-node/RDMA and optional Cosmos GPU behavior have not been revalidated.