Remove dead modules, measurement artifacts, and stale references - #77
Draft
qywu wants to merge 1 commit into
Draft
Remove dead modules, measurement artifacts, and stale references#77qywu wants to merge 1 commit into
qywu wants to merge 1 commit into
Conversation
Sweeps the repository for files with no consumer, using an AST import graph over all tracked Python (583 src modules) with entry points resolved, including `-m` invocations documented in README and docs. Dead source: - `xorl.ops.group_gemm.utils` (whole subpackage): the `__init__` is empty, nothing outside the package imports it, and its only consumers were two commented-out `@pretuned(...)` decorators in `kernel/group_gemm.py`. No `config/` tree exists for `load_all_configs` to read, so the autotuning path could not have resolved anyway. The kernels use `triton.autotune`. Drops the commented decorators with it. - `weight_sync.backends.nccl_simple`: `create_backend()` has no branch for it and `handler.py` rejects `sync_method="nccl_simple"`, so the backend was unreachable. The handler docstring and the weight-sync README both advertised it as supported; corrected. - `ops.loss.utils`, `utils.recompute_utils`, `utils.manual_cuda_timing`, `utils.model_utils`, `utils.distillation_utils`, `data.prepare.file_lock_loader`: no caller in src, tests, examples, scripts, or configuration. Their last consumers were tests removed by the suite consolidation, which deliberately left production APIs in place. Measurement artifacts, which record a past A/B rather than protecting a current contract: - `benchmarks/r3_replay_staging_benchmark.py`, a one-off harness for comparing two source trees via PYTHONPATH. The correctness it sampled is covered by the routing-replay handler and MoE routing-replay tests. - `certification/benchmark_vocab_parallel_ce.py`, `certification/glm52/benchmark_sparse_mla_backward.py`, and `certification/qwen3_30b/compare_lora_qlora.py`. The remaining certification entries stay: multi-node DeepEP/AllToAll parity and the opt-in official GLM-5.2 FP8 inventory cannot run in CI and have no in-suite equivalent. - Two scratch training configs (`timing_test.yaml`, `bench_qlora_compile.yaml`) that set a 2- and 20-step budget and write to benchmark output directories. - `examples/server/password_memorization/run_train_and_infer.py`, whose docstring documents a `sft_and_sample.py` entry point that no longer exists and which the directory README supersedes with the unified `run_password_test.py`. Also removes the test-audit ledger and the alternate sglang manifest, and repairs the references they leave behind: the pre-commit large-file exclusion, the triage workflow step that wrote to the ledger, and the combined-environment install instructions in the README and two docs pages. Six test filenames cited by the testing and RMSNorm-contract docs no longer exist; they now name live tests. Verified: compileall and check_public_tree pass, ruff output is identical to the pre-change baseline, and pytest collects the same 1979 tests with the same five pre-existing import errors from packages absent in this environment.
Broly Security ScanNote ✅ Clean scan Note Re-scan this PR anytime with
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sweeps the repository for files with no consumer. Candidates came from an AST import graph over all tracked Python (583
src/modules) with entry points resolved, including the-minvocations documented in the README and docs — a plain grep misses those and would have flaggedxorl.cli.trainas dead.23 files removed, ~18,600 lines. No production behavior changes.
Dead source
xorl.ops.group_gemm.utils(whole subpackage, 8 files). The__init__is empty, nothing outside the package imports it, and its only consumers were two commented-out@pretuned(...)decorators inkernel/group_gemm.py. Noconfig/tree exists forload_all_configsto read, so the autotuning path could not have resolved even if uncommented; the kernels usetriton.autotune. The commented decorators go with it.weight_sync.backends.nccl_simple.create_backend()has no branch for it andhandler.pyrejectssync_method="nccl_simple", so the backend was unreachable. The handler docstring and the weight-sync README both advertised it as supported — corrected.ops.loss.utils,utils.recompute_utils,utils.manual_cuda_timing,utils.model_utils,utils.distillation_utils,data.prepare.file_lock_loader. No caller in src, tests, examples, scripts, or configuration. Their last consumers were tests removed by the suite consolidation, which deliberately left production APIs in place —manual_cuda_timingstill claimed the trainer drained it, but nothing calleddrain_manual_cuda_timing.Measurement artifacts
These record a past A/B rather than protecting a current contract:
benchmarks/r3_replay_staging_benchmark.py— a one-off harness for comparing two source trees viaPYTHONPATH. The correctness it sampled is covered bytests/server/runner/test_routing_replay_handler.pyandtests/models/test_moe_routing_replay.py.certification/benchmark_vocab_parallel_ce.py,certification/glm52/benchmark_sparse_mla_backward.py,certification/qwen3_30b/compare_lora_qlora.py.timing_test.yaml,bench_qlora_compile.yaml) with 2- and 20-step budgets writing to benchmark output directories.examples/server/password_memorization/run_train_and_infer.py, whose docstring documents asft_and_sample.pyentry point that no longer exists and which the directory README supersedes with the unifiedrun_password_test.py.The remaining
certification/entries stay: multi-node DeepEP/AllToAll parity and the opt-in official GLM-5.2 FP8 inventory cannot run in CI and have no in-suite equivalent.Ledger, manifest, and the references they left behind
test_audit_decisions.jsonandpyproject.sglang.tomlare removed, along with what pointed at them: the pre-commit large-file exclusion, the triage workflow step that wrote to the ledger, and the combined-environment install instructions in the README and two docs pages. Six test filenames cited by the testing and RMSNorm-contract docs no longer exist; they now name live tests.Verification
compileallandscripts/check_public_tree.pypass.ruffoutput is byte-identical to the pre-change baseline (117 findings, all pre-existing; 112 in the ruff-excluded vendoredops/quacktree).pytest tests --collect-onlygives the same 1979 tests and the same five pre-existing import errors, all from packages absent in this environment (pytest_asyncio,tilelang.tileop.gemm.registry,transformers.models.nemotron_h).Deliberately left alone
Three groups are unreferenced but documented as intentional, so this PR does not touch them. Each is worth a separate decision:
src/xorl/ops/quack/— vendored fork of Quack 0.4.1, globally ruff-excluded. ~11 unreachable modules (~3,000 lines), but pruning upstream library code complicates re-vendoring.glm5_kernels/indexer.py+tilelang_indexer_bwd.py,dsv4/kernel/tilelang_indexer.py+tilelang_indexer_bwd.py, ~590 lines).glm5_kernels/__init__.pyexplains they are carried because "miles'SparseMLA/IndexerFunctionautograd wrappers reference them at class-definition time." That rationale rests onmiles_plugins, which nothing in this repo imports, so it may now be moot.gdn_decode_prep.py+solve_tril_decode.py(362 lines) — "Nothing routes here by default; callers opt in", and no opt-in caller exists, tests included.Follow-ups this surfaces
pyproject.sglang.tomldrops the only pinned recipe for a combined xorl + xorl-client + xorl-sglang environment. The docs now say to install the submodule into a separate environment, but the exact PyTorch 2.11 / CUDA 13 / Triton 3.6 pin set is gone. If thexorl-sglangsubmodule is also being retired, that should be a follow-up.XORL_GLM5_SPLIT_SPARSE_MLA_BWDand its legacy split path inops/glm5_kernels/tilelang_sparse_mla_bwd.pylost their only consumer with the sparse-MLA benchmark.TEST_TRIAGE.md(5,550 lines) is now a campaign log whose ledger is gone, whileCONTRIBUTING.mdstill makes its process normative.scripts/opd/run_opd_pipeline.py(1,424 lines) targets a k8s Job with no manifests in this repo; its only in-repo protection is a payload-shape test.