Skip to content

Remove dead modules, measurement artifacts, and stale references - #77

Draft
qywu wants to merge 1 commit into
mainfrom
cleanup/remove-dead-files-and-artifacts
Draft

Remove dead modules, measurement artifacts, and stale references#77
qywu wants to merge 1 commit into
mainfrom
cleanup/remove-dead-files-and-artifacts

Conversation

@qywu

@qywu qywu commented Aug 21, 2026

Copy link
Copy Markdown
Member

Sweeps the repository for files with no consumer. Candidates came from an AST import graph over all tracked Python (583 src/ modules) with entry points resolved, including the -m invocations documented in the README and docs — a plain grep misses those and would have flagged xorl.cli.train as dead.

23 files removed, ~18,600 lines. No production behavior changes.

Dead source

  • xorl.ops.group_gemm.utils (whole subpackage, 8 files). The __init__ is empty, nothing outside the package imports it, and its only consumers were two commented-out @pretuned(...) decorators in kernel/group_gemm.py. No config/ tree exists for load_all_configs to read, so the autotuning path could not have resolved even if uncommented; the kernels use triton.autotune. The commented decorators go with it.
  • weight_sync.backends.nccl_simple. create_backend() has no branch for it and handler.py rejects sync_method="nccl_simple", so the backend was unreachable. The handler docstring and the weight-sync README both advertised it as supported — corrected.
  • ops.loss.utils, utils.recompute_utils, utils.manual_cuda_timing, utils.model_utils, utils.distillation_utils, data.prepare.file_lock_loader. No caller in src, tests, examples, scripts, or configuration. Their last consumers were tests removed by the suite consolidation, which deliberately left production APIs in place — manual_cuda_timing still claimed the trainer drained it, but nothing called drain_manual_cuda_timing.

Measurement artifacts

These record a past A/B rather than protecting a current contract:

  • benchmarks/r3_replay_staging_benchmark.py — a one-off harness for comparing two source trees via PYTHONPATH. The correctness it sampled is covered by tests/server/runner/test_routing_replay_handler.py and tests/models/test_moe_routing_replay.py.
  • certification/benchmark_vocab_parallel_ce.py, certification/glm52/benchmark_sparse_mla_backward.py, certification/qwen3_30b/compare_lora_qlora.py.
  • Two scratch training configs (timing_test.yaml, bench_qlora_compile.yaml) with 2- and 20-step budgets writing to benchmark output directories.
  • examples/server/password_memorization/run_train_and_infer.py, whose docstring documents a sft_and_sample.py entry point that no longer exists and which the directory README supersedes with the unified run_password_test.py.

The remaining certification/ entries stay: multi-node DeepEP/AllToAll parity and the opt-in official GLM-5.2 FP8 inventory cannot run in CI and have no in-suite equivalent.

Ledger, manifest, and the references they left behind

test_audit_decisions.json and pyproject.sglang.toml are removed, along with what pointed at them: the pre-commit large-file exclusion, the triage workflow step that wrote to the ledger, and the combined-environment install instructions in the README and two docs pages. Six test filenames cited by the testing and RMSNorm-contract docs no longer exist; they now name live tests.

Verification

  • compileall and scripts/check_public_tree.py pass.
  • ruff output is byte-identical to the pre-change baseline (117 findings, all pre-existing; 112 in the ruff-excluded vendored ops/quack tree).
  • pytest tests --collect-only gives the same 1979 tests and the same five pre-existing import errors, all from packages absent in this environment (pytest_asyncio, tilelang.tileop.gemm.registry, transformers.models.nemotron_h).

Deliberately left alone

Three groups are unreferenced but documented as intentional, so this PR does not touch them. Each is worth a separate decision:

  1. src/xorl/ops/quack/ — vendored fork of Quack 0.4.1, globally ruff-excluded. ~11 unreachable modules (~3,000 lines), but pruning upstream library code complicates re-vendoring.
  2. Indexer backward kernels (glm5_kernels/indexer.py + tilelang_indexer_bwd.py, dsv4/kernel/tilelang_indexer.py + tilelang_indexer_bwd.py, ~590 lines). glm5_kernels/__init__.py explains they are carried because "miles' SparseMLA/IndexerFunction autograd wrappers reference them at class-definition time." That rationale rests on miles_plugins, which nothing in this repo imports, so it may now be moot.
  3. gdn_decode_prep.py + solve_tril_decode.py (362 lines) — "Nothing routes here by default; callers opt in", and no opt-in caller exists, tests included.

Follow-ups this surfaces

  • Removing pyproject.sglang.toml drops the only pinned recipe for a combined xorl + xorl-client + xorl-sglang environment. The docs now say to install the submodule into a separate environment, but the exact PyTorch 2.11 / CUDA 13 / Triton 3.6 pin set is gone. If the xorl-sglang submodule is also being retired, that should be a follow-up.
  • XORL_GLM5_SPLIT_SPARSE_MLA_BWD and its legacy split path in ops/glm5_kernels/tilelang_sparse_mla_bwd.py lost their only consumer with the sparse-MLA benchmark.
  • TEST_TRIAGE.md (5,550 lines) is now a campaign log whose ledger is gone, while CONTRIBUTING.md still makes its process normative.
  • scripts/opd/run_opd_pipeline.py (1,424 lines) targets a k8s Job with no manifests in this repo; its only in-repo protection is a payload-shape test.

Sweeps the repository for files with no consumer, using an AST import
graph over all tracked Python (583 src modules) with entry points
resolved, including `-m` invocations documented in README and docs.

Dead source:

- `xorl.ops.group_gemm.utils` (whole subpackage): the `__init__` is empty,
  nothing outside the package imports it, and its only consumers were two
  commented-out `@pretuned(...)` decorators in `kernel/group_gemm.py`.
  No `config/` tree exists for `load_all_configs` to read, so the
  autotuning path could not have resolved anyway. The kernels use
  `triton.autotune`. Drops the commented decorators with it.
- `weight_sync.backends.nccl_simple`: `create_backend()` has no branch for
  it and `handler.py` rejects `sync_method="nccl_simple"`, so the backend
  was unreachable. The handler docstring and the weight-sync README both
  advertised it as supported; corrected.
- `ops.loss.utils`, `utils.recompute_utils`, `utils.manual_cuda_timing`,
  `utils.model_utils`, `utils.distillation_utils`,
  `data.prepare.file_lock_loader`: no caller in src, tests, examples,
  scripts, or configuration. Their last consumers were tests removed by
  the suite consolidation, which deliberately left production APIs in
  place.

Measurement artifacts, which record a past A/B rather than protecting a
current contract:

- `benchmarks/r3_replay_staging_benchmark.py`, a one-off harness for
  comparing two source trees via PYTHONPATH. The correctness it sampled is
  covered by the routing-replay handler and MoE routing-replay tests.
- `certification/benchmark_vocab_parallel_ce.py`,
  `certification/glm52/benchmark_sparse_mla_backward.py`, and
  `certification/qwen3_30b/compare_lora_qlora.py`. The remaining
  certification entries stay: multi-node DeepEP/AllToAll parity and the
  opt-in official GLM-5.2 FP8 inventory cannot run in CI and have no
  in-suite equivalent.
- Two scratch training configs (`timing_test.yaml`,
  `bench_qlora_compile.yaml`) that set a 2- and 20-step budget and write
  to benchmark output directories.
- `examples/server/password_memorization/run_train_and_infer.py`, whose
  docstring documents a `sft_and_sample.py` entry point that no longer
  exists and which the directory README supersedes with the unified
  `run_password_test.py`.

Also removes the test-audit ledger and the alternate sglang manifest, and
repairs the references they leave behind: the pre-commit large-file
exclusion, the triage workflow step that wrote to the ledger, and the
combined-environment install instructions in the README and two docs
pages. Six test filenames cited by the testing and RMSNorm-contract docs
no longer exist; they now name live tests.

Verified: compileall and check_public_tree pass, ruff output is identical
to the pre-change baseline, and pytest collects the same 1979 tests with
the same five pre-existing import errors from packages absent in this
environment.
@broly-code-security-scanner

Copy link
Copy Markdown

Broly Security Scan

Note

Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

Ask in #security-engineering   Powered by Together AI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant