Skip to content

Cancel a PR's orphaned workflow runs when the PR is closed - #415

Merged
hmgaudecker merged 2 commits into
mainfrom
ci/cancel-closed-pr-runs
Aug 3, 2026
Merged

hmgaudecker merged 2 commits into
mainfrom
ci/cancel-closed-pr-runs

Conversation

@hmgaudecker

@hmgaudecker hmgaudecker commented Aug 3, 2026 •

Copy link
Copy Markdown
Member

Closes the gap that blocked the GPU queue for 84 minutes today.

The problem

GitHub does not cancel a PR's queued or in-progress runs when the PR is closed.
main.yml's concurrency guard cannot cover this:

concurrency:
  group: ${{ github.head_ref || github.run_id }}
  cancel-in-progress: true

That only supersedes a run when a newer run appears on the same head ref. Closing a PR
is not a new run, so it never fires.

With a single self-hosted GPU runner, the orphans become head-of-line blockers.

Observed 2026-08-03. PR #413 (grid-search-ez → reachability) was open from 11:21:03
to 11:23:48 — 2m45s. Its run 30809182760 outlived it by ~84 minutes, sitting in the GPU
queue in front of #414 until the runs were cancelled by hand at 12:45. #390's run also had
to be force-cancelled: its 32-bit GPU job ignored the ordinary cancel endpoint entirely.

The fix

A reaper on pull_request_target: [closed] that cancels runs for that head ref.

Five things worth reviewing:

  • Scope: PRs closed without merging. closed also fires on merge, so the job is
    guarded by if: github.event.pull_request.merged == false. Merged PRs' runs are
    deliberately left to finish.
  • pull_request_target, not pull_request — the token is read-only for fork PRs on
    pull_request, so the reaper would silently no-op on exactly the PRs least likely to be
    cleaned up by hand. This is only safe because the job never checks out or executes PR
    code; it reads the head ref and calls the REST API. Do not add a checkout step.
  • No injection surface. On a fork PR the head ref is attacker-controlled, so it reaches
    jq via $ENV and gh via -f (which also URL-encodes it) — never a shell word or a raw
    query string. Branch names may legally contain & and #.
  • Live-PR guard. A head ref can back more than one PR — retargeting a stack branch is
    the obvious case here. The job bails if any other open PR still points at the same head
    repo + ref, so closing one cannot cancel a live one's runs.
  • Bounded escalation to force-cancel. 120s grace, then force. Without this the
    workflow would not have fixed today's incident: the graceful cancel was ignored.

Known cost of exempting merged PRs

This is documented at the guard so it isn't rediscovered the hard way. benchmark-pr runs
in concurrency group gpu-benchmarks-<head_ref>, while the post-merge benchmark-main run
uses gpu-benchmarks-main — different groups, so merging does not supersede the PR's
benchmark run.
With timeout-minutes: 14400 (10 days) and a while nvidia-smi … sleep 60
wait-for-GPU loop, a benchmark run started on a PR can hold the benchmark runner well past
the merge. Cancel it by hand if that happens.

Verification

Ran against live API state before committing:

check result
-f branch="feat/dcegm" filters correctly (slash in ref) matches, 3 runs
$ENV works in gh's jq (gojq) yes
live-PR guard on reachability (#414 open) still_open=1 → skips
run query for grid-search-ez finds 30809182760
that run's status when #413 closed queued → inside the filter, would have been reaped
set -e survives [ … ] && continue survives
bash -n on the extracted run: block OK
yamllint, check-github-workflows, full prek pass

Not exercised end-to-end on GitHub — pull_request_target runs the version of the workflow
on the base branch, so it does nothing until this lands on main. The first real test
is the next PR closed without merging.

GitHub does not cancel queued or in-progress runs when a PR is closed, and
`main.yml`'s concurrency group only supersedes a run when a *newer* run appears
on the same head ref -- closing a PR is not a new run, so it never fires.

The orphaned runs then sit in the single self-hosted GPU queue and block every
later PR behind them. Observed on 2026-08-03: PR #413 was open for 2m45s, and
its run outlived it by ~84 minutes, holding the head of the GPU queue in front
of #414 until the runs were cancelled by hand.

Notes on the implementation:

- `pull_request_target`, not `pull_request`, so the token is writable for fork
  PRs too. Safe only because nothing here checks out or executes PR code.
- The head ref reaches jq via `$ENV` and gh via `-f`, never a shell word or a
  raw URL -- on a fork PR it is attacker-controlled.
- It skips if any *other* open PR still points at the same head repo + ref, so
  retargeting a stack branch cannot cancel a live PR's runs.
- Bounded escalation to `force-cancel`: a long-running self-hosted job can
  ignore the graceful cancel, which is what kept the GPU queue blocked above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QipFjBYxb5aPnmZE76cxhi
@read-the-docs-community

read-the-docs-community Bot commented Aug 3, 2026 •

Copy link
Copy Markdown

@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.39%. Comparing base (56f244d) to head (dcd135e).
⚠️ Report is 190 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #415      +/-   ##
==========================================
- Coverage   98.30%   90.39%   -7.92%     
==========================================
  Files          42      170     +128     
  Lines        2415    15153   +12738     
==========================================
+ Hits         2374    13697   +11323     
- Misses         41     1456    +1415     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

`closed` also fires on merge, so the first version cancelled a merged PR's runs
too. HMG's call is to let those finish, so the job is now guarded by

    if: github.event.pull_request.merged == false

The cost is documented at the guard rather than left to be rediscovered: a
`benchmark-pr` run sits in concurrency group `gpu-benchmarks-<head_ref>` while
the post-merge `benchmark-main` run sits in `gpu-benchmarks-main`, so merging
does not supersede it. Combined with `timeout-minutes: 14400` (10 days) and the
"wait for GPU to be free" spin loop, a benchmark run started on a PR can hold
the benchmark runner well past the merge and needs cancelling by hand.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QipFjBYxb5aPnmZE76cxhi
@hmgaudecker
hmgaudecker merged commit a000698 into main Aug 3, 2026
3 of 10 checks passed
@hmgaudecker
hmgaudecker deleted the ci/cancel-closed-pr-runs branch August 3, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant