ci: run the whole test suite against the real server on every PR (#1) - #18
Merged
Conversation
One workflow (.github/workflows/ci.yml), and it runs the WHOLE suite against the real server on every pull request and every push to main (plus manual dispatch) — no reduced/fast variant: - server job: rake install from the checkout (a --user-install, so both Gem.user_dir/bin and Gem.bindir go on PATH — the two only coincide on root/rbenv setups), mruby HEAD + the reflection VM via mruby-lsp-setup against a fixture workspace, then ALL of test/overlay (including the live-VM pin mruby_semantics_test.rb), the conformance replays over real LSP stdio, and test/consistency driven by a real LSP client (headless Neovim + clangd) against the real server. - editor job: the extension suite in a real VS Code extension host (xvfb-run -a npm test) and the debug-adapter protocol tests. Three fixes this workflow needed to be honest, folded in: - The extension tests now run in a real VS Code extension host (official @vscode/test-cli, fixture workspaces under test/fixtures/, nothing about the vscode API mocked). The prior hand-mocked vscode harness had drifted from the product on three axes — a missing context.extension.packageJSON, a 7th declared command it never asserted, and a set-up-detection protocol it no longer matched — every drift shipping silently because nothing real loaded the extension. - The conformance replay clients (test/conformance/replay*.py) answer the server's unsandboxed-consent dialog over window/showMessageRequest (ours_launch.rb runs the server without the launcher, so on Linux it is always unconfined and the fail-closed consent gate always fires), and correctly route server-initiated requests instead of misreading them as responses. - mruby HEAD changed bare module_function to match CRuby (public singleton copy + private instance method, ending at a bare visibility verb) — caught by the live-VM pin test/overlay/mruby_semantics_test.rb exactly as designed, now modeled by the overlay and re-pinned. Conformance replays are structured to never need a workflow edit again: each replay script computes its own real exit code (sys.exit(0 if fail==0 and passed>0 else 1) — the passed>0 guard so a wrong RLSRC or empty fixture glob reads as red, not an accidental green), and test/conformance/replays/ holds one small runner script per feature (mirroring test/overlay's own for-loop pattern). ci.yml just loops over that directory; adding a conformance feature means dropping a script there, never touching this workflow. Verified end-to-end, repeatedly: a full local rehearsal (mruby HEAD build through the real reflection VM, all replays, both consistency suites) before every push, and green runs on the GitHub Actions runners themselves.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.github/workflows/ci.yml), running the whole suite against the real server on every pull request and every push tomain(plus manual dispatch) — no reduced/fast variant.rake installfrom the checkout, mruby HEAD + the reflection VM viamruby-lsp-setup, then all oftest/overlay(including the live-VM pinmruby_semantics_test.rb), the conformance replays over real LSP stdio, andtest/consistencyunder a real LSP client (headless Neovim + clangd).xvfb-run -a npm test) and the debug-adapter protocol tests.vscodemodule that had drifted from the product; the conformance replay clients answer the server's unsandboxed-consent dialog instead of dying at the first request; mruby HEAD's changedmodule_functionsemantics are re-pinned in the overlay.test/conformance/replays/holds one small runner script per feature —ci.ymljust loops over that directory.Test plan
workflow_dispatchon this branch)