ci(ff): support feature freeze automation - #7339
Open
fr4nc1sc0-r4m0n wants to merge 25 commits into
Open
Conversation
Resolve go.mod/go.sum conflicts by keeping main dependency versions and retaining go-git/go-github deps required for release automation.
fr4nc1sc0-r4m0n
added
Team:Elastic-Agent-Control-Plane
fr4nc1sc0-r4m0n
added
Team:Elastic-Agent-Control-Plane
|
| Status | Scan Engine |  Critical |
 High |
 Medium |
 Low |
Total (0) |
|---|---|---|---|---|---|---|
| Open Source Security | 0 | 0 | 0 | 0 | See details | |
| Licenses | 0 | 0 | 0 | 0 | See details |
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.
Contributor
|
This pull request does not have a backport label. Could you fix it @fr4nc1sc0-r4m0n? 🙏
|
fr4nc1sc0-r4m0n
added
backport-active-9
8 tasks
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Allow release workflows to be safely retriggered by no-oping when the target version is already applied, reusing existing branches and open PRs, and skipping empty commits.
Move release workflow logic into a dedicated package aligned with elastic-agent and beats, leaving thin mage release:* wrappers in magefile.go.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Enable gomoddirectives replace-local so the dev-tools submodule replace required by mage release imports passes golangci-lint.
The root go.mod replaces dev-tools with a local path. Docker layer caching must include dev-tools/go.mod and go.sum so go mod download can resolve the replaced module.
fleet-server publishes version tags but no GitHub Releases, so runMajorMinor failed resolving LatestRelease for minor versions.
Contributor
Author
Test MinorClick here to expandDRY_RUN=false CURRENT_RELEASE="9.6.0" mage release:runMajorMinor
=== Starting Major/Minor Release Workflow ===
Resolved LatestRelease from elastic/fleet-server: 9.4.4
Checked out branch: main
Verified CURRENT_RELEASE=9.6.0 matches 9.6.0 on branch main
Creating release branch: 9.6
Already on branch: main
Created branch: 9.6
Checked out branch: 9.6
--- Preparing PR-A: backport rule + version 9.7.0 on main ---
Checked out branch: main
Created branch: ff-prep-main-9.6.0
Checked out branch: ff-prep-main-9.6.0
Added backport rule for 9.6 to .mergify.yml
Updated version to 9.7.0 in version/version.go
Created commit: c0b0a7f275a902c7fc6961504e0468b054d187ba
--- Preparing PR-B: ff-release 9.6.0 on 9.6 ---
Checked out branch: 9.6
Created branch: ff-release-9.6.0
Checked out branch: ff-release-9.6.0
Version already set to 9.6.0 in version/version.go
No changes to commit
--- Preparing PR-D: next patch 9.6.1 on 9.6 ---
Checked out branch: 9.6
Created branch: ff-prep-next-patch-9.6.1
Checked out branch: ff-prep-next-patch-9.6.1
Updated version to 9.6.1 in version/version.go
Created commit: 5ba76c11eeff9dc1c0cf34b6f44925158a5ac19a
Checked out branch: 9.6
Pushed branch 9.6 to remote: origin
Checked out branch: ff-prep-main-9.6.0
Pushed branch ff-prep-main-9.6.0 to remote: origin
Created label "merge:1-ff-day"
Added labels to #2: [release impact:critical backport-9.6 skip-changelog Team:Automation merge:1-ff-day]
Created PR #2: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/2
Checked out branch: ff-release-9.6.0
No new commits on ff-release-9.6.0 compared to 9.6; skipping push and PR creation
Checked out branch: ff-prep-next-patch-9.6.1
Pushed branch ff-prep-next-patch-9.6.1 to remote: origin
Created label "merge:4-after-release"
Added labels to #3: [release Team:Automation skip-changelog merge:4-after-release]
Created PR #3: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/3
=== Major/Minor Release Workflow Complete ===
Release branch created: 9.6
PR 1: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/2 (open)
PR 2: skipped (no related open/merged PR for ff-release-9.6.0 → 9.6)
PR 3: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/3 (open)
Note: PR-C (main docs) is omitted for fleet-server (no docs/test-env updates)PR ComparisonFormer bot flow opened 3 PRs per minor FF (version bump + backport rule + next patch). The new
|
Contributor
Author
Test PatchClick here to expand$ DRY_RUN=false CURRENT_RELEASE="9.4.5" mage release:runPatch
=== Starting Patch Release Workflow ===
Checked out branch: 9.4
Verified CURRENT_RELEASE=9.4.5 matches 9.4.5 on branch 9.4
--- Preparing PR-A: ensure version 9.4.5 on 9.4 ---
Already on branch: 9.4
Checked out branch: patch-release-9.4.5
Version already set to 9.4.5 in version/version.go
No changes to commit
--- Preparing PR-D: next patch 9.4.6 on 9.4 ---
Checked out branch: 9.4
Checked out branch: ff-prep-next-patch-9.4.6
Version already set to 9.4.6 in version/version.go
No changes to commit
Checked out branch: patch-release-9.4.5
No new commits on patch-release-9.4.5 compared to 9.4; skipping push and PR creation
Checked out branch: ff-prep-next-patch-9.4.6
Pushed branch ff-prep-next-patch-9.4.6 to remote: origin
Added labels to #4: [release Team:Automation skip-changelog merge:4-after-release]
Created PR #4: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/4
=== Patch Release Workflow Complete ===
PR 1: skipped (no related open/merged PR for patch-release-9.4.5 → 9.4)
PR 2: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/4 (open)PR ComparisonFormer bot flow opened 1 PR per patch (next-patch version bump only; fleet-server has no docs/manifests patch PR). The new
|
Contributor
|
This pull request is now in conflicts. Could you fix it @fr4nc1sc0-r4m0n? 🙏 |
fr4nc1sc0-r4m0n
marked this pull request as ready for review
July 24, 2026 11:13
Resolve NOTICE.txt and NOTICE-fips.txt conflicts by keeping main's portable $GOMODCACHE paths; retain go.mod replace for dev-tools and main's elastic-agent-libs bump.
Contributor
|
This pull request is now in conflicts. Could you fix it @fr4nc1sc0-r4m0n? 🙏 |
Resolve version-bump pipeline conflicts by keeping mage release targets and main's master DRA steps; prefer main's dev-tools dependency versions.
Contributor
Author
|
@elastic/elastic-agent-control-plane can you take a look, please? |
There was a problem hiding this comment.
Pull request overview
This PR migrates Fleet Server feature-freeze and patch release automation from Makefile/external scripts to Mage-driven Go tooling, implemented as a nested Go module under dev-tools/mage/release/ with a small CLI wrapper to keep go-git / go-github dependencies out of the root module.
Changes:
- Add a nested
dev-tools/mage/releaseGo module +fleet-releaseCLI implementing the FF/patch workflows and release issue tracker updates. - Add root
mage release:*wrapper targets to invoke the nested CLI, and wire the workflows into the Buildkite version-bump pipeline. - Update Docker build caching and lint config to accommodate the new
dev-toolsmodule wiring, and add release documentation.
Reviewed changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| RELEASE.md | New operator-facing docs for Mage-based release workflows and env vars. |
| magefile.go | Adds release:* Mage namespace wrappers that call the nested CLI. |
| go.mod | Adds a local replace for the dev-tools submodule. |
| Dockerfile.fips | Copies dev-tools/go.mod + go.sum for reproducible module resolution during builds. |
| Dockerfile.build | Copies dev-tools/go.mod + go.sum for dependency caching/validation in build images. |
| Dockerfile | Copies dev-tools/go.mod + go.sum to support local module replacement during go mod download. |
| dev-tools/mage/release/workflows.go | Implements feature-freeze and patch workflows (branch/PR orchestration). |
| dev-tools/mage/release/workflows_test.go | Unit tests for labels and PR body content. |
| dev-tools/mage/release/version.go | Semver parsing + “latest release before current” selection helper. |
| dev-tools/mage/release/version_test.go | Tests for latest-release selection behavior. |
| dev-tools/mage/release/release.go | File allowlisting + version file read/update helpers. |
| dev-tools/mage/release/release_test.go | Tests for version and Mergify updates + idempotency. |
| dev-tools/mage/release/README.md | Developer/operator reference for the nested release module. |
| dev-tools/mage/release/mergify.go | Logic to append backport rules to .mergify.yml. |
| dev-tools/mage/release/issue.go | Creates/updates a release checklist issue and merges discovered PR links. |
| dev-tools/mage/release/issue_test.go | Tests for issue body generation/merge behavior. |
| dev-tools/mage/release/go.sum | Dependency checksums for the nested release module. |
| dev-tools/mage/release/go.mod | Declares the nested release module and its dependencies. |
| dev-tools/mage/release/github.go | GitHub API wrapper for PR/issue/label/release discovery operations. |
| dev-tools/mage/release/github_test.go | HTTP-test-server based unit tests for GitHub client behavior. |
| dev-tools/mage/release/git.go | go-git based helpers for branch creation, commits, and pushes. |
| dev-tools/mage/release/git_test.go | Tests for git helper behaviors and naming. |
| dev-tools/mage/release/config.go | Loads/derives release config from env and resolves LatestRelease when needed. |
| dev-tools/mage/release/config_test.go | Tests for env config loading and version derivation. |
| dev-tools/mage/release/cmd/fleet-release/main.go | CLI entrypoint that locates repo root and dispatches workflow commands. |
| .golangci.yml | Allows local replace directives (to support the new module structure). |
| .buildkite/pipeline.version-bump.yaml | Switches version-bump automation to run the new mage release:* workflows. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // enter the main fleet-server go.mod / NOTICE.txt. | ||
| module github.com/elastic/fleet-server/dev-tools/mage/release | ||
|
|
||
| go 1.26.5 |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 27 changed files in this pull request and generated 5 comments.
Suppressed comments (2)
dev-tools/mage/release/go.mod:5
- The nested release module uses a different Go toolchain version than the rest of the repo (.go-version and other go.mod files use 1.26.7). Keeping these aligned avoids confusing local/CI behavior when running
go run -C dev-tools/mage/release ....
go 1.26.5
dev-tools/mage/release/issue.go:246
- mergeReleaseIssueBody rebuilds the body as
before + ## PRs + newPRBlock, which drops anything that appears after the "## PRs" section in an existing issue (manual notes, additional sections, etc.). That makes the update non-idempotent and can cause unintended data loss.
before := body[:idx]
oldPRBlock := strings.TrimPrefix(body[idx+len(prSection):], "\n")
newPRBlock := formatPRChecklist(allURLs, existingChecked)
oldURLs := keys(extractPRCheckboxes(oldPRBlock))
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 27 changed files in this pull request and generated 3 comments.
Suppressed comments (2)
dev-tools/mage/release/go.mod:5
- This nested module declares
go 1.26.5, which is inconsistent with the repo toolchain (.go-versionis 1.26.7, and the root/dev-tools modules usego 1.26.7). Keeping the go version aligned avoids unexpected toolchain/format churn when runninggo mod tidyacross modules.
go 1.26.5
dev-tools/mage/release/mergify.go:32
UpdateMergifyvalidateslen(parts) < 2, but the error message says it expectsX.Y.Z. This function is called withcfg.ReleaseBranch(e.g.9.6), so the message should reflect the accepted inputs (X.Y or X.Y.Z) to avoid confusing failures.
parts := strings.Split(version, ".")
if len(parts) < 2 {
return fmt.Errorf("invalid version format: %s (expected X.Y.Z)", version)
}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 27 changed files in this pull request and generated 3 comments.
Suppressed comments (1)
dev-tools/mage/release/mergify.go:27
- UpdateMergify unmarshals .mergify.yml into map[string]interface{} and then marshals it back, which will drop comments and likely reformat/reorder large parts of the file. In practice this makes PR-A very noisy and risks producing a YAML shape/style Mergify doesn’t accept (e.g., branch names like 8.16/9.5 may be emitted unquoted). Consider switching to an implementation that preserves the original file (e.g., append a properly-indented rule text block to the existing pull_request_rules list, or manipulate a yaml.Node tree and force branch scalars to be !!str/double-quoted) so the diff is limited to the new rule only.
var config map[string]interface{}
if err := yaml.Unmarshal(content, &config); err != nil {
return fmt.Errorf("failed to parse %s: %w", mergifyFile, err)
}
Comment on lines
+235
to
+255
| const prSection = "## PRs" | ||
| idx := strings.Index(body, prSection) | ||
| if idx < 0 { | ||
| updated := strings.TrimRight(body, "\n") + "\n\n" + prSection + "\n\n" + formatPRChecklist(allURLs, existingChecked) | ||
| return updated, true | ||
| } | ||
|
|
||
| before := body[:idx] | ||
| oldPRBlock := strings.TrimPrefix(body[idx+len(prSection):], "\n") | ||
| newPRBlock := formatPRChecklist(allURLs, existingChecked) | ||
|
|
||
| oldURLs := keys(extractPRCheckboxes(oldPRBlock)) | ||
| if !sameStringSet(oldURLs, allURLs) || normalizePRSection(oldPRBlock) != normalizePRSection(newPRBlock) { | ||
| changed = true | ||
| } | ||
| if !changed { | ||
| return existingBody, false | ||
| } | ||
|
|
||
| updated := strings.TrimRight(before, "\n") + "\n\n" + prSection + "\n\n" + newPRBlock | ||
| return updated, true |
Comment on lines
+401
to
+418
| func ensurePatchCurrentReleaseMatchesBranch(repo *GitRepo, cfg *ReleaseConfig) error { | ||
| if err := repo.CheckoutBranch(cfg.ReleaseBranch); err != nil { | ||
| return err | ||
| } | ||
| branchVersion, err := ReadFleetVersion() | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if branchVersion != cfg.CurrentRelease { | ||
| return fmt.Errorf( | ||
| "CURRENT_RELEASE=%s does not match version on branch %s (%s in version/version.go); "+ | ||
| "set CURRENT_RELEASE to the version already on the release branch (the patch being released)", | ||
| cfg.CurrentRelease, cfg.ReleaseBranch, branchVersion, | ||
| ) | ||
| } | ||
| fmt.Printf("Verified CURRENT_RELEASE=%s matches %s on branch %s\n", cfg.CurrentRelease, branchVersion, cfg.ReleaseBranch) | ||
| return nil | ||
| } |
Comment on lines
+81
to
+83
| if err := g.CheckoutBranch(baseBranch); err != nil { | ||
| return fmt.Errorf("failed to checkout base branch %s: %w", baseBranch, err) | ||
| } |
Contributor
Author
|
buildkite test this |
Contributor
TL;DRBuildkite Remediation
Investigation detailsRoot CauseThe failing step ends in Terraform destroy output and then a generic shell exit status 1, with no preceding Fleet Server test assertion, panic, or compile/runtime error. The tail of the failing log shows only teardown of an Elastic Cloud deployment (
This indicates the failure occurred in the CI script lifecycle around teardown/command handling, not in Fleet Server product logic. Evidence
Verification
Follow-up
What is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the problem this PR solves?
Fleet Server release automation still relies on Makefile-based scripts and external CLI tools. This PR migrates the feature-freeze and patch release workflows to pure Go Mage targets so release steps can run without depending on hub, gh, sed, yq, or Python.
This continues the work from #6584 and aligns the process with beats (#51831) and elastic-agent (#15433).
How does this PR solve the problem?
dev-tools/mage/release/withcmd/fleet-releaseCLI sogo-git/go-githubstay out of the rootgo.mod/NOTICE.txtmage release:*wrappers invoke the nested CLIrunNextRelease(next-patch prep is PR-D / patch PR-B)DRY_RUN=truemode for safe local testingRELEASE.mdanddev-tools/mage/release/README.mdWorkflow alignment
fleet-server.mak)prepare-major-minor-release+create-branch-major-minor-release(+ next-release steps)mage release:runMajorMinorX.Y; PR-A (main: mergify + next minor); PR-D (release branch: next patch)prepare-patch-release+create-prs-patch-releasemage release:runPatchFeature freeze (
CURRENT_RELEASEalready onmain)X.Yfrommainff-prep-main-{CURRENT}→mainmerge:1-ff-dayff-prep-next-patch-{NEXT}→X.Ymerge:4-after-releasePatch (
CURRENT_RELEASEalready on release branch)ff-prep-next-patch-{NEXT}→X.Ymerge:4-after-releaseFiles updated
version/version.go—DefaultVersion.mergify.yml— backport rule (PR-A)Idempotency
UpdateVersionUpdateMergifyCommitAllCreatePRHow to test this PR locally
Discard local workflow changes after review with
git reset --hard HEAD.Checklist
./changelog/fragmentsusing the changelog toolRelated issue: https://github.com/elastic/observability-robots/issues/3404
Validations
Minor releases
Former bot flow opened 3 PRs per minor FF (version bump + backport rule + next patch). The new
runMajorMinorflow groups bump + mergify into PR-A and opens PR-D for the next patch (PR-C omitted; no docs/test-env; PR-B is a no-op when the release branch already has the correct version):main+ backport rulePatch releases
Former bot flow opened 1 PR per patch (next-patch version bump only; fleet-server has no docs/manifests patch PR). The new
runPatchflow opens PR-D for the next patch (PR-A ensure-version is usually a no-op and was skipped in this run):