[Benchmark] Add DFlash vs baseline accuracy/perf sweep scripts by TianHao65 · Pull Request #13 · apinge/sglang

TianHao65 · 2026-08-06T02:39:57Z

Add five standalone benchmark drivers under benchmark/dflash/ that launch servers across (attention_backend, tp_size) configs and run a workload for each (concurrency, num_questions) setting, reporting DFlash throughput and acceptance length against a baseline run.

bench_dflash_gsm8k_sweep.py GSM8K
bench_dflash_math500_sweep.py MATH500, scored by \boxed{...} string
equivalence (conservative lower bound)
bench_dflash_humaneval_sweep.py HumanEval pass@1, sandboxed subprocess
bench_dflash_mbpp_sweep.py MBPP pass@1, sandboxed subprocess
bench_dflash_mtbench_sweep.py MT-Bench, performance only (no judge)

For the code benchmarks, test execution runs after the timed generation region so it never contaminates throughput or accept-length numbers.

Launch environment matches launch_qwen3.5-397B-fp8_tp8_prefix_cache_DFlash.sh and is applied via setdefault before importing torch/sglang, so explicit caller exports still win.

These are benchmark scripts, not CI tests -- they are long-running by design.

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

Add five standalone benchmark drivers under benchmark/dflash/ that launch servers across (attention_backend, tp_size) configs and run a workload for each (concurrency, num_questions) setting, reporting DFlash throughput and acceptance length against a baseline run. - bench_dflash_gsm8k_sweep.py GSM8K - bench_dflash_math500_sweep.py MATH500, scored by \boxed{...} string equivalence (conservative lower bound) - bench_dflash_humaneval_sweep.py HumanEval pass@1, sandboxed subprocess - bench_dflash_mbpp_sweep.py MBPP pass@1, sandboxed subprocess - bench_dflash_mtbench_sweep.py MT-Bench, performance only (no judge) For the code benchmarks, test execution runs after the timed generation region so it never contaminates throughput or accept-length numbers. Launch environment matches launch_qwen3.5-397B-fp8_tp8_prefix_cache_DFlash.sh and is applied via setdefault before importing torch/sglang, so explicit caller exports still win. These are benchmark scripts, not CI tests -- they are long-running by design.

apinge · 2026-08-06T02:50:15Z

No CI required since this PR is only for benchmarks. I have manually canceled the run.

Copilot

Pull request overview

This PR adds standalone benchmark drivers under benchmark/dflash/ to sweep DFLASH vs baseline across multiple (attention_backend, tp_size) server configs and (concurrency, num_questions) workloads, reporting throughput and speculative acceptance-length; the code-eval benchmarks also compute pass@1 in a post-timing phase.

Changes:

Added GSM8K and MATH500 sweep scripts that measure throughput/accept-length and compute basic string-based accuracy.
Added HumanEval and MBPP sweep scripts that measure throughput/accept-length and compute pass@1 via sandboxed subprocess execution after timing.
Added an MT-Bench sweep script that measures performance only (no judge), using first-turn prompts as single-turn requests.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
benchmark/dflash/bench_dflash_gsm8k_sweep.py	GSM8K DFLASH vs baseline sweep with throughput + accept-length + numeric-answer accuracy.
benchmark/dflash/bench_dflash_math500_sweep.py	MATH500 DFLASH vs baseline sweep with throughput + accept-length + boxed-answer string equivalence accuracy.
benchmark/dflash/bench_dflash_humaneval_sweep.py	HumanEval DFLASH vs baseline sweep with throughput + accept-length + pass@1 via subprocess test execution (post-timing).
benchmark/dflash/bench_dflash_mbpp_sweep.py	MBPP DFLASH vs baseline sweep with throughput + accept-length + pass@1 via subprocess assertion execution (post-timing).
benchmark/dflash/bench_dflash_mtbench_sweep.py	MT-Bench DFLASH vs baseline sweep focused on performance only (no judge).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sammysun0711

@TianHao65, please help to check coplot review comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

sammysun0711 · 2026-08-10T13:56:31Z

+    "SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE": "1",
+    "SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB": "1",
+    "USE_AITER_COMM": "1",
+    "AITER_MOE_SMALL_BATCH": "1",


We do not need this environment variable on latest qwen3_5_v0.5.15_dflash branch, please remove it for simplicity.

sammysun0711 · 2026-08-10T13:56:51Z

+    "AITER_MOE_PADDING_SIZE": "192",
+    "HIP_GDN_SORT_IDX_BS": "32768",


We do not need this environment variable on latest qwen3_5_v0.5.15_dflash branch, please remove it for simplicity.

sammysun0711 · 2026-08-10T13:59:36Z

+    device_sm = get_device_sm()
+    attention_backends = _filter_attention_backends(
+        attention_backends, device_sm=device_sm
+    )


Why should we run nvidia flash attention backend?

sammysun0711 · 2026-08-10T14:01:39Z

+def _filter_attention_backends(backends: list[str], *, device_sm: int) -> list[str]:
+    if not (80 <= device_sm <= 90):
+        backends = [b for b in backends if b != "fa3"]
+    if device_sm < 100:
+        backends = [b for b in backends if b not in ("fa4", "trtllm_mha")]
+    return backends or ["flashinfer"]


What is output from this function call on rocm platform?

sammysun0711 · 2026-08-10T14:07:23Z

+    "AITER_MOE_PADDING_SIZE": "192",
+    "HIP_GDN_SORT_IDX_BS": "32768",


To be cleanup. please all coments in benchmark/dflash/bench_dflash_gsm8k_sweep.py

sammysun0711 · 2026-08-10T14:07:34Z

+    "SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE": "1",
+    "SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB": "1",
+    "USE_AITER_COMM": "1",
+    "AITER_MOE_SMALL_BATCH": "1",


To be cleanup. please all coments in benchmark/dflash/bench_dflash_gsm8k_sweep.py

sammysun0711 · 2026-08-10T14:11:54Z

+        "--max-running-requests",
+        str(args.max_running_requests),
+        "--attention-backend",
+        backend,


Should add --linear-attn-backend aiter.

apinge assigned sammysun0711 Aug 6, 2026

sammysun0711 requested a lite review from Copilot August 7, 2026 08:10

Copilot started reviewing on behalf of sammysun0711 August 7, 2026 08:11 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

sammysun0711 requested changes Aug 7, 2026

View reviewed changes

TianHao65 and others added 10 commits August 10, 2026 11:37

Potential fix for pull request finding

1db5d23

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

60a2b49

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

83ca2a1

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

3e43a2b

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

7c743ba

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

9443b0d

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

ecf04d4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

845bce2

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

b9b9a56

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

4428930

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

apinge requested a review from sammysun0711 August 10, 2026 04:10

sammysun0711 reviewed Aug 10, 2026

View reviewed changes

		"AITER_MOE_PADDING_SIZE": "192",
		"HIP_GDN_SORT_IDX_BS": "32768",

Conversation

TianHao65 commented Aug 6, 2026

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

Uh oh!

apinge commented Aug 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sammysun0711 left a comment

Choose a reason for hiding this comment

Uh oh!

sammysun0711 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sammysun0711 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sammysun0711 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sammysun0711 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sammysun0711 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sammysun0711 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sammysun0711 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants