From 3601f6e0a468fd187ccc8f4bb65b8640b55a366a Mon Sep 17 00:00:00 2001 From: Tim Smith Date: Sat, 29 Aug 2026 18:50:23 -0700 Subject: [PATCH] test: add Test Kitchen integration suites running against fog mocks Until now the only automated coverage was RSpec. Nothing ever loaded the driver through Test Kitchen itself, so a plugin that failed to register, a Driver API mismatch, a file left out of the gemspec, or an exception on the real create path could all reach a release with a green build. This adds ten suites in kitchen.yml that drive the driver's actual create and destroy code through Test Kitchen, against fog-openstack's mock backend instead of a live cloud. They need no credentials and no OpenStack, and the whole set runs in about 40 seconds. The suites cover a plain boot, image and flavor selection by name, by /regex/ and by id, a network resolved by ref, a floating IP taken from a pool and a pinned one, a prefixed server name, the optional server settings, inline cloud_config, and user_data from a file. Fog has to be mocked before Test Kitchen instantiates the driver, so test/fog_mock.rb is required ahead of the kitchen executable rather than from kitchen.yml, which keeps kitchen.yml an ordinary example file. A real cloud is still required for anything only a cloud decides -- quotas, scheduling, real networking, SSH -- and that is not something hosted CI can do. CONTRIBUTING.md now says so plainly, and lists the three paths fog's mocks do not model, so nobody mistakes this for full coverage. Signed-off-by: Tim Smith --- .github/workflows/lint.yml | 21 +++++++ AGENTS.md | 14 +++-- CONTRIBUTING.md | 52 +++++++++++++--- Rakefile | 7 +++ kitchen.yml | 104 ++++++++++++++++++++++++++++++++ test/fog_mock.rb | 32 ++++++++++ test/integration/cloud-init.yml | 6 ++ 7 files changed, 223 insertions(+), 13 deletions(-) create mode 100644 kitchen.yml create mode 100644 test/fog_mock.rb create mode 100644 test/integration/cloud-init.yml diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 042bc85b..f99d55dd 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -7,3 +7,24 @@ name: 'Lint, Unit & Integration Tests' jobs: lint-unit: uses: test-kitchen/.github/.github/workflows/lint-unit.yml@main + + integration: + name: 'Integration / Ruby ${{ matrix.ruby }}' + runs-on: ubuntu-latest + needs: lint-unit + strategy: + fail-fast: false + matrix: + ruby: + - '3.1' + - '3.4' + steps: + - uses: actions/checkout@v7 + - uses: ruby/setup-ruby@v1 + with: + ruby-version: ${{ matrix.ruby }} + bundler-cache: true + # Drives the driver's real create/destroy code through Test Kitchen + # against fog-openstack's mock backend, so it needs no credentials and + # no OpenStack deployment. See CONTRIBUTING.md. + - run: bundle exec rake integration diff --git a/AGENTS.md b/AGENTS.md index 7b0b2071..36e85ac4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,12 +28,13 @@ kitchen-openstack is a Test Kitchen driver for OpenStack. It provisions and dest ```bash bundle install -bundle exec rake # runs tests + style (default) -bundle exec rake test # unit tests only (RSpec) -bundle exec rake style # Cookstyle lint -bundle exec rake quality # style -bundle exec rake yard # render YARD docs to doc/ (not CI-gated) -bundle exec rake yard_stats # list undocumented methods +bundle exec rake # runs tests + style (default) +bundle exec rake test # unit tests only (RSpec) +bundle exec rake integration # Test Kitchen suites against fog-openstack mocks +bundle exec rake style # Cookstyle lint +bundle exec rake quality # style +bundle exec rake yard # render YARD docs to doc/ (not CI-gated) +bundle exec rake yard_stats # list undocumented methods ``` ## Conventions @@ -47,4 +48,5 @@ bundle exec rake yard_stats # list undocumented methods - `verify_partial_doubles` is on. Fog *model* classes take `instance_double`; Fog *service* objects cannot, because Fog defines their methods dynamically at instantiation - Unit tests never sleep, hit the network, or read outside a `Dir.mktmpdir`. The one exemption is `openstack_version_spec.rb`, which reads the gemspec and the Release Please manifest to catch version drift, and skips when they are absent. `ENV` is replaced with an `OS_*`-free hash, and the clouds.yaml specs additionally pin `Dir.pwd`, `Dir.home` and `/etc/openstack`, so neither a developer's OpenStack environment nor their real `clouds.yaml`/`secure.yaml` can leak in - SimpleCov reports to `coverage/` with no enforced threshold +- Integration suites live in `kitchen.yml` and run against fog-openstack's mock backend; `test/fog_mock.rb` must be required before the `kitchen` executable (`ruby -Itest -r fog_mock -S kitchen test`). No credentials or cloud needed - Release automation via Release Please — version bumps go in `lib/kitchen/driver/openstack_version.rb` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f653a0e8..5c866f5a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,14 +29,17 @@ bundle install ## Rake tasks ```bash -bundle exec rake # tests + lint (default) -bundle exec rake test # unit tests only -bundle exec rake style # Cookstyle lint -bundle exec rake yard # render docs to doc/ -bundle exec rake yard_stats # list undocumented methods +bundle exec rake # tests + lint (default) +bundle exec rake test # unit tests only +bundle exec rake integration # Test Kitchen suites against fog's mocks +bundle exec rake style # Cookstyle lint +bundle exec rake yard # render docs to doc/ +bundle exec rake yard_stats # list undocumented methods ``` -`rake test` runs the `unit` task, and `rake quality` runs `style`. +`rake test` runs the `unit` task, and `rake quality` runs `style`. The default +task deliberately leaves `integration` out so that the common case stays fast; +CI runs it as its own job. To run a single spec file: @@ -77,9 +80,44 @@ undocumented. Neither is enforced in CI, but new methods should come with docs. When you add or change a configuration option, update the configuration reference in `README.md` as well. +## Integration tests + +`bundle exec rake integration` runs the suites in `kitchen.yml` through Test +Kitchen itself. They need no credentials and no OpenStack deployment: +`test/fog_mock.rb` puts Fog into mock mode before the driver is loaded, so +`create` and `destroy` run their real code against fog-openstack's in-memory +Nova, Neutron and Cinder. + +Fog has to be mocked before Test Kitchen instantiates the driver, which is why +the rake task requires the file ahead of the `kitchen` executable rather than +doing it from `kitchen.yml`: + +```bash +bundle exec ruby -Itest -r fog_mock -S kitchen test +bundle exec ruby -Itest -r fog_mock -S kitchen test network-ref-cirros +bundle exec ruby -Itest -r fog_mock -S kitchen diagnose --all +``` + +What this catches that the unit tests cannot: the driver failing to load or +register, a Driver API version mismatch, a file missing from the gemspec, a +config key that no longer survives `finalize_config!`, and any exception on the +real create/destroy path — each of which the unit tests can miss because they +never go through Test Kitchen's own plugin loading and instance lifecycle. + +What it cannot catch: anything only a real cloud decides. Quotas, scheduling, +whether the image actually boots, real networking, and SSH are all outside a +mock. There is no way to run those in hosted CI — they need a live OpenStack — +so they stay a manual step, below. + +A few paths are not covered as suites because fog's mocks do not model them: +allocating a *new* floating IP (the mock's response carries no address), +Cinder volumes (mock volumes never reach `available`), and reading an address +back off a named network (mock servers are not attached to the mock networks). +Those are covered by the unit tests, and by the manual pass. + ## Manual testing -The unit tests never contact a cloud, so changes that touch instance creation, +The tests above never contact a cloud, so changes that touch instance creation, networking, or credential resolution should also be exercised against a real OpenStack deployment. diff --git a/Rakefile b/Rakefile index 18cf778b..45e7f6d4 100644 --- a/Rakefile +++ b/Rakefile @@ -6,6 +6,13 @@ RSpec::Core::RakeTask.new(:unit) desc "Run all test suites" task test: [:unit] +desc "Run the Test Kitchen integration suites against fog-openstack's mocks" +task :integration do + # test/fog_mock.rb has to be loaded before Test Kitchen instantiates the + # driver, hence -r rather than anything inside kitchen.yml. + sh "bundle exec ruby -Itest -r fog_mock -S kitchen test" +end + desc "Run the unit tests with coverage reporting to coverage/" task :coverage do ENV["COVERAGE"] = "1" diff --git a/kitchen.yml b/kitchen.yml new file mode 100644 index 00000000..4801b78d --- /dev/null +++ b/kitchen.yml @@ -0,0 +1,104 @@ +--- +# Test Kitchen integration suites for this driver. +# +# These run against fog-openstack's mock backend rather than a real cloud, so +# they need no credentials and no OpenStack deployment. Put Fog into mock mode +# by requiring test/fog_mock.rb ahead of the kitchen executable: +# +# bundle exec rake integration +# bundle exec ruby -Itest -r fog_mock -S kitchen test network-ref-cirros +# +# Every suite drives the driver's real create and destroy code through Test +# Kitchen itself. What that catches, and what it cannot, is written up in +# CONTRIBUTING.md -- in short, anything a live cloud would reject (quotas, +# scheduling, real networking, SSH) still needs the manual pass described +# there. +# +# The image and flavor names below are the ones fog-openstack's mock Nova +# ships with. + +driver: + name: openstack + image_ref: cirros-0.3.0-x86_64-blank + flavor_ref: 512 server + openstack_auth_url: http://keystone.example.com:5000/v3 + openstack_username: kitchen + openstack_api_key: kitchen + openstack_domain_id: default + glance_cache_wait_timeout: 60 + +provisioner: + name: dummy + +transport: + name: dummy + +verifier: + name: dummy + +platforms: + - name: cirros + +suites: + # Boot and tear down with nothing but an image and a flavor set. + - name: default + + # image_ref and flavor_ref given as /regex/ rather than as exact names, + # which takes a different path through find_matching. + - name: image-flavor-regex + driver: + image_ref: /^cirros/ + flavor_ref: /512/ + + # image_id and flavor_id, which are used verbatim with no lookup at all. + # These are the ids the mock Nova assigns to the image and flavor named + # above; they are fixed constants in fog-openstack's mock data. + - name: image-flavor-id + driver: + image_ref: null + flavor_ref: null + image_id: 0e09fbd6-43c5-448a-83e9-0d3d05f9747e + flavor_id: '2' + + # A network resolved by name and turned into a nics entry. + - name: network-ref + driver: + network_ref: network_1 + + # Takes an already-allocated, unattached address out of a pool and + # associates it, then reads it back as the hostname. + - name: floating-ip-pool + driver: + floating_ip_pool: nova + + # A specific address, which short-circuits address resolution entirely. + - name: floating-ip + driver: + floating_ip: 192.168.27.130 + + # Server naming: a user-supplied prefix plus a random suffix. + - name: server-name-prefix + driver: + server_name_prefix: kitchen-ci + + # The optional server settings, all of which are resolved by optional_config. + - name: server-options + driver: + availability_zone: nova + security_groups: + - default + metadata: + kitchen: integration + config_drive: true + + # Inline cloud-config, which the driver renders to user_data itself. + - name: cloud-config + driver: + cloud_config: + packages: + - htop + + # user_data read from a file on disk. + - name: user-data + driver: + user_data: test/integration/cloud-init.yml diff --git a/test/fog_mock.rb b/test/fog_mock.rb new file mode 100644 index 00000000..2126c198 --- /dev/null +++ b/test/fog_mock.rb @@ -0,0 +1,32 @@ +# frozen_string_literal: true + +# +# Copyright:: (C) 2026, Oregon State University +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Puts Fog into mock mode before Test Kitchen loads the driver, so the +# integration suites in kitchen.yml run the driver's real create/destroy code +# against fog-openstack's in-memory Nova, Neutron and Cinder instead of a live +# cloud. +# +# Required ahead of the `kitchen` executable rather than from kitchen.yml, so +# that kitchen.yml stays an ordinary example file: +# +# bundle exec ruby -Itest -r fog_mock -S kitchen test +# +# See CONTRIBUTING.md for what this does and does not cover. + +require "fog/openstack" + +Fog.mock! diff --git a/test/integration/cloud-init.yml b/test/integration/cloud-init.yml new file mode 100644 index 00000000..dd286775 --- /dev/null +++ b/test/integration/cloud-init.yml @@ -0,0 +1,6 @@ +#cloud-config +# Fixture for the user-data integration suite. The mock Nova accepts the +# contents unread; the point of the suite is that the driver finds the file, +# reads it, and puts it on the server definition. +packages: + - htop