Add a beginner guide to Beaker acceptance testing#368
Conversation
f360527 to
81f3977
Compare
e2d53f1 to
ed253c5
Compare
5322444 to
c76cb92
Compare
Expand the DevKit "Acceptance Testing with Beaker" page from an 8-line stub into a full beginner-friendly guide. It covers: - prerequisites: a hypervisor, a current Ruby (via rbenv/rvm/mise), and the system_tests Bundler group, plus a macOS/Apple Silicon local-run note - wiring voxpupuli-acceptance into a module (Gemfile, spec_helper, Rakefile) - anatomy of a test: the apply_manifest two-run idempotency idiom with a worked chrony example that mirrors the real puppet-chrony test - choosing a platform (setfiles), running the suite, the common BEAKER_* variables, debugging a failure, running in CI, and customizing nodes Beginner-facing terms (catalog, gem, hypervisor, rake, fixtures, apply_manifest, setfile, the package resource type) are linked or glossed on first use. The commands and the openvox8 collection name were validated end to end by running the suite against real OpenVox containers (almalinux9-64 and debian12-64, 4 examples / 0 failures) and in GitHub Actions CI. Closes OpenVoxProject#304 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Michael Harp <mike@mikeharp.com>
The acceptance testing guide was reachable from the nav but not mentioned on the DevKit components overview. Add a Beaker bullet alongside the other testing tools, linking to the guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Michael Harp <mike@mikeharp.com>
c76cb92 to
0024476
Compare
binford2k
left a comment
There was a problem hiding this comment.
This is fine the way it is, but I do have a few suggestions to make the whole set feel more cohesive.
| If Docker reports an image platform mismatch, export `DOCKER_DEFAULT_PLATFORM=linux/amd64`, pull the base image once with that platform selected, and retry. | ||
|
|
||
| Second, you need a current Ruby. | ||
| The Ruby that ships with macOS and some Linux distributions is too old, and using it makes `bundle install` fail with a confusing dependency error rather than a clear "your Ruby is too old" message. |
There was a problem hiding this comment.
maybe worth calling out that macOS has deprecated system ruby, even though they've not actually removed it yet.
There was a problem hiding this comment.
actually, this comment makes me realize that the setup page should talk more about ruby versions. Filed #382
| Beaker defaults to Docker, which is the easiest way to get started, so install it and make sure your user can run containers. | ||
| [Podman](https://podman.io/), [Vagrant](https://www.vagrantup.com/), and [libvirt](https://libvirt.org/) are also supported if you prefer them. | ||
|
|
||
| If you're running locally on macOS, two extra wrinkles come up. Neither affects Linux CI runners, which already provide the standard socket and architecture. |
There was a problem hiding this comment.
Could we maybe put these into a <details> disclosure widget or something? It's not super clear on first read which parts are the "two extra wrinkles".
Either that, or move it to the end of the prereqs section.
| Install a 3.x Ruby, then run `ruby --version` in the module directory and confirm it reports 3.x before you go further; the [DevKit setup guide](setup.html) covers this in more detail. | ||
|
|
||
| Finally, you need the acceptance gems installed. | ||
| A module's `Gemfile` sorts its gems into named groups so that, for example, continuous integration can install only what each job needs. |
There was a problem hiding this comment.
I think the with/without discussion should be moved to setup.md and this section should only say something like
You'll need the
system_testsbundler group, but you can omitdevelopmentandreleaseif you want to keep things lean.
filed #383
| That second run is the heart of acceptance testing, because it proves _idempotency_. | ||
| A manifest is idempotent when applying it more than once is safe: the first run brings the system to the desired state, and every run after that makes no changes because the system already matches. | ||
| Idempotency is a core property of well-written Puppet code, and a manifest that keeps making changes on every run is a bug. | ||
| Catching that here is one of the most valuable things an acceptance test does. |
There was a problem hiding this comment.
Is it worth mentioning the modules which require two runs to reach idempotency? (the ones that install something used to generate a fact that is then used on the next run to specify state)
Or is that a "by the time you have to do this, you'll know enough to figure it out" thing?
- Wrap the macOS-only prerequisites in a <details> disclosure whose summary names the two wrinkles (Docker socket, CPU architecture), so it's clear on first read which parts apply and that Linux CI skips them. - Link Apple's Catalina release notes noting the system Ruby is deprecated and slated for removal from a future macOS. - Add a note that a few modules legitimately need two runs to converge, so the two-run idempotency check has a known exception. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Michael Harp <mike@mikeharp.com>
What
Expands the DevKit "Acceptance Testing with Beaker" page from an 8-line stub (which only linked to
voxpupuli-acceptance) into a full beginner-friendly guide.Closes #304.
Contents
system_testsBundler group, plus a macOS/Apple Silicon local-run note. The common newcomer traps are called out explicitly.voxpupuli-acceptanceinto a module (thesystem_testsGemfile group,spec_helper_acceptance.rb, and theRakefile).apply_manifesttwo-run idempotency idiom (with idempotency defined plainly), a workedchronyexample that mirrors the realpuppet-chronyacceptance test, and the shared-example shortcuts.BEAKER_*environment-variable reference, debugging a failure, running in CI (including a pointer to the sharedgha-puppetreusable workflow that most Vox Pupuli modules use, with a note on the modulesync-managedci.yml), and customizing nodes.catalog,gem,hypervisor,rake,fixtures,apply_manifest,setfile, thepackageresource type) are linked or glossed on first use.Validation
Every command in the guide was validated end to end:
voxpupuli-acceptanceinto a real module and ranrake beakerin GitHub Actions (Docker,almalinux9-64,openvox8), green.voxpupuli/puppet-chronymodule on macOS + Docker:bundle install,puppet-metadata setfiles, andbundle exec rake beaker.The worked example was run as written against real OpenVox containers on both
almalinux9-64anddebian12-64→ 4 examples, 0 failures each. Local testing also pinned down two adjustments the worked example needs: chrony won't apply cleanly inside a Docker container withoutoptions => '-F 0 -x'(which disable clock-stepping and the seccomp filter), and the service is namedchronydon RHEL-family vschronyon Debian-family, so the example selects it fromos.family.That validation is also where the guide's details were pinned down: the
openvox8collection name (vs. upstreampuppet8), thepuppet-metadata setfilescommand (the oldersetfilesis now a deprecated alias), and the macOS/Apple Silicon and Ruby-version prerequisites.Notes
The page lives in
_ecosystem_8xand is surfaced via the existing_latestsymlink. The nav entry is renamed from "Acceptance Testing" to "Acceptance Testing with Beaker" to match the page title.