feat (image-test): run LISA test suites locally via 'image test'#292
Draft
liunan-ms wants to merge 1 commit into
Draft
feat (image-test): run LISA test suites locally via 'image test'#292liunan-ms wants to merge 1 commit into
liunan-ms wants to merge 1 commit into
Conversation
Previously LISA-type suites were rejected by 'azldev image test' and only served as metadata for external orchestration. This adds a local runner that generates a LISA runbook from a suite's configured test cases, boots the image in a QEMU VM, and runs the tests. - Extend LisaConfig with a framework git source, test-cases, pip pre-install/extras, and extra-args; validate that the framework ref is a full commit SHA and that test-cases is non-empty. - Generate a qemu-platform runbook that inlines the image path and an ephemeral admin key, with keep_environment=no so VMs are torn down. - Auto-generate and clean up an ephemeral admin SSH key pair per run. - Require the image to be qcow2 (reject other formats) before booting. - Regenerate CLI docs, JSON schema, and scenario snapshots.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enables running LISA-based image test suites locally via azldev image test, expanding the existing test-suite model (previously only pytest was runnable) to support generating a LISA runbook, booting the image via QEMU, and executing LISA against it.
Changes:
- Extend test-suite configuration with an optional
[lisa]subtable (LisaConfig) including framework git source + test cases and related options, plus validation of the framework ref and presence of test cases. - Add a local LISA runner that clones the framework repo at a pinned commit, creates/reuses a per-suite venv, generates a qemu-platform runbook, generates an ephemeral SSH keypair, and invokes LISA.
- Regenerate schema, CLI docs, and scenario snapshots to reflect the new configuration surface.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| schemas/azldev.schema.json | Generated JSON schema updates to include GitSourceConfig and LisaConfig. |
| scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap | Snapshot refresh for schema generation output (container mode). |
| scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap | Snapshot refresh for schema generation output (non-container mode). |
| internal/projectconfig/testsuite.go | Adds LisaConfig, GitSourceConfig, and new validation logic for LISA suites. |
| internal/projectconfig/testsuite_test.go | Unit tests for new LISA validation behavior. |
| internal/projectconfig/loader_test.go | Loader test updated to parse and validate a LISA suite config. |
| internal/app/azldev/cmds/image/test.go | Routes lisa suites to the new local runner and updates help text. |
| internal/app/azldev/cmds/image/lisarunner.go | New implementation: clone framework, venv setup, runbook generation, qcow2 enforcement, and LISA invocation. |
| internal/app/azldev/cmds/image/lisarunbook.go | New runbook YAML model + generator for QEMU platform. |
| internal/app/azldev/cmds/image/lisarunbook_internal_test.go | Unit tests for runbook YAML generation. |
| docs/user/reference/cli/azldev_image_test.md | Regenerated CLI docs describing LISA suite behavior. |
Comment on lines
+106
to
+108
| // TestCases lists the LISA test case names to run. They are combined (joined with '|') | ||
| // into the criteria of a runbook that azldev generates and runs. | ||
| TestCases []string `toml:"test-cases,omitempty" json:"testCases,omitempty" jsonschema:"title=Test cases,description=LISA test case names to run; combined into a generated runbook's criteria"` |
Comment on lines
+95
to
+102
| // Generate a runbook from the configured test cases and write it into the framework | ||
| // tree so its relative includes resolve. | ||
| runbookPath, err := writeGeneratedRunbook( | ||
| env, frameworkDir, suiteConfig.Name, lisaConfig.TestCases, options.ImagePath, adminKeyPath, | ||
| ) | ||
| if err != nil { | ||
| return err | ||
| } |
Comment on lines
+238
to
+239
| shortSHA := source.Ref[:shortSHALength] | ||
| repoDir := filepath.Join(baseDir, category, shortSHA) |
Comment on lines
+265
to
+270
| } else { | ||
| slog.Info("Reusing existing git repo", | ||
| slog.String("path", repoDir), | ||
| slog.String("ref", source.Ref), | ||
| ) | ||
| } |
liunan-ms
changed the title
Contributor
|
Hey there and thank you for opening this pull request! 👋🏼 We require pull request titles to follow the Conventional Commits specification and it looks like your proposed title needs to be adjusted. Details: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Previously LISA-type test suites were rejected by
azldev image testand only served as metadata for external orchestration. This adds a local runner that generates a LISA runbook from a suite's configured test cases, boots the image in a QEMU VM, and runs the tests.Changes
LisaConfigwith a framework git source,test-cases, pip pre-install/extras, andextra-args; validate that the framework ref is a full commit SHA and thattest-casesis non-empty.keep_environment=noso VMs are torn down after the run.Testing
mage unit— passmage check all— pass (golangci-lint, editorconfig, staticcheck, licenses, ruff, pyright)mage scenarioUpdate— snapshots regeneratedUsage in microsoft/azurelinux#18042