Repository navigation
fix: extended the e2e suites and fixed the defects they exposed - #20
Merged
Merged
Conversation
scripts/e2e.sh was one linear flow over the database, filesystem and S3-bucket sources. The coverage gaps still open - alert delivery, multi-job, TLS, encryption, arm64 - each need their own receivers and servers, and CI should run them as parallel legs, not one long job. * scripts/e2e.sh is now the runner: argument parsing, the image override file, the shared helpers, the image build, teardown and the summary. It runs the suites named as arguments or in E2E_SUITES, in order, on one stack; "all" runs every suite. * scripts/e2e/engines.sh holds the existing round trips unchanged. It is the default suite, so "bash scripts/e2e.sh" and the three CI legs run exactly what they ran before. * On a failure the logs of every service are printed, not only the database and MinIO ones, so suites with other services are covered.
Every postgres round trip ran as "app", the superuser the postgres image creates, so nothing showed that an application's own role - the usual credential in a consumer stack - can back up and restore its database. A superuser hides every missing privilege. The engines suite now creates app_owner (no SUPERUSER, CREATEDB, CREATEROLE or BYPASSRLS, asserted from pg_roles) and a database it owns, seeds it as that role - a plain table, a partitioned table and grants for the restricted app_runtime role - and runs the engine as app_owner with keep_acl: * backup, uploaded to MinIO; * restore into the emptied database; * restore over the live partitioned table after rows changed and the runtime role's grants were revoked: rows and grants come back. The password is built at runtime (rand_secret, now shared by the suites), never written into the repository.
MariaDB was only backed up by the application user that owns every
object, and MySQL had the only least-privilege check. Events were in no
round trip, although the dump runs with --events and a restore must
recreate them.
* The seed of both servers adds a disabled, future-dated event; every
MariaDB and MySQL restore check now expects the trigger, the
procedure and the event back ("1|1|1|1").
* MariaDB: bk_min backs up with SELECT, SHOW VIEW, TRIGGER and EVENT on
the database plus SHOW CREATE ROUTINE (MariaDB 11.3+) - no LOCK
TABLES, no global privilege - and the application user restores the
snapshot with trigger, procedure and event. bk_exec sees the
procedure only through EXECUTE: its backup must end in error with
"insufficient privileges" instead of leaving the routine out.
* The grants under test are printed with SHOW GRANTS in the log.
The CI legs run this on MariaDB 13 (compose), 11.4 (lts) and the
newest release; MySQL keeps its existing least-privilege checks, now
with the event.
No CI run had ever delivered an alert: every channel was unit-tested
against a recording transport or a loopback SMTP stub, so the wire
format, the TLS modes, the SMTP login, the level filter and the
escaping of run data were never checked where a receiver sees them.
The new alerts suite (scripts/e2e/alerts.sh, scripts/e2e_alerts.py)
runs the engine against:
* sink - the engine image's python recording every HTTP alert: the
signed webhook, Teams, Slack, Discord, ntfy and Healthchecks;
* mail-starttls / mail-smtps - Mailpit with STARTTLS required on 587
and implicit TLS on 465, login required on both.
Three jobs with the levels all / warnings / errors and every channel
run a success, a warning and an error scenario. The warning and the
error carry run data - a file name and a missing path made of markup,
a Slack mention and Markdown. Checked on arrival: the HMAC-SHA256
signature over the raw sorted-keys body, the payload, the Teams
adaptive and MessageCard shapes and colors, the Slack/Discord summary,
ntfy title and bearer token, the Healthchecks /fail ping, which level
delivered which status, and for every mail the subject, the SMTP
login, the run figures and the run data - HTML-escaped in the HTML
part, verbatim in the text part, absent from every chat channel.
Two more runs: SMTPS without trusting the CA must refuse to send while
the webhook still delivers, and a webhook that never answers must not
hold up the run or the Slack post after it (killed after 180 s).
TLS uses a CA generated per run with openssl (scripts/e2e.sh,
.e2e-pki/, mounted read-only at /pki); the engine trusts it through
SSL_CERT_FILE. Webhook secret, ntfy token and SMTP password are built
at runtime and reach the engine through ${VAR} interpolation.
Multi-job configs - job-scoped snapshot ids, per-job retention and prune, the per-job healthcheck - were only unit-tested, and the scheduler daemon itself never ran in CI: every round trip used --now. The new multijob suite starts the daemon with two jobs that share the data dir and a cron schedule (on_startup plus every minute), so the scheduler starts both in the same second, and checks: * the second round's ids are <same second>_alpha / _beta - no collision, every id names its job; * alpha (count 1) keeps one snapshot, beta (count 3) two: each job prunes only its own; prune --job beta --keep 1 leaves alpha alone; * a run record per job, both success; * docker stop drains the daemon (SIGTERM handler) and it exits 0; * verify of both ids, and restore of beta's job-scoped id without --job lands in job beta's path; * the healthcheck: healthy with both jobs fresh; alpha alone stale under its own healthcheck_max_age_hours; after a run in which beta fails and alpha succeeds later, unhealthy naming beta only. The suite works in a data dir of its own (/data/multijob), so it can share a stack with the other suites.
The verify_tls and ca_bundle settings of the S3 destination and source were only tested against a local Python HTTPS stub that answers head_bucket. No run had uploaded to, listed, hydrated from or mirrored a real S3 server behind TLS, and every MinIO in CI spoke plain HTTP. The new tls suite runs MinIO with HTTPS only (minio-tls, the runtime CA's certificate from scripts/e2e.sh) and checks: * default verification refuses the private CA with CERTIFICATE_VERIFY_FAILED; the snapshot stays local, run in warning; * with ca_bundle the upload succeeds; after the local copy is deleted, list shows the snapshot off-site only, restore hydrates it over TLS and restores the file, and verify hydrates it again (sha256 OK); * verify_tls false uploads and logs the "verification is DISABLED" warning naming the endpoint; * an s3 source mirroring that bucket fails without ca_bundle and completes with it. MinIO's root password is built at runtime; the suite keeps its own data dir (/data/tls).
Encryption was only unit-tested with a fake subprocess: no run had encrypted a real archive with age or gpg, verified it or restored it, and the keys were never more than argv strings. The new encryption suite generates its keys in the run with the image's own tools (volume e2e-keys at /keys, nothing in the repo): an age identity, and a gpg key pair whose public key is imported into a second keyring that holds nothing else - imported, never trusted - as on a backup host that must not be able to decrypt. Checked: * age: the snapshot exists only as <id>.tar.gz.age with the age header, verify passes, restore with encryption.identity_file brings the file back; a restore without an identity ends with "restore finished with errors" naming identity_file, no traceback; * gpg: backed up with the public-key-only keyring, stored only as <id>.tar.gz.gpg (a public-key encrypted packet), verify passes, restore with the secret keyring brings the file back; * failures: a recipient age cannot parse and a key the keyring lacks each store <id>.tar.gz UNENCRYPTED, log the error and end the run in warning. The suite keeps its own data dir (/data/encryption).
The CS-MinIO images (minio, minio-init) are published for linux/amd64 only, so the engines suite could not run on an arm64 host at all. With E2E_MINIO=0 the suite starts the databases without MinIO and gives every job an S3 destination with an empty bucket, which the engine skips as "not configured" - the documented local-only path, so the configs stay the same. The MinIO archive checks and the S3-bucket source round trips are reported as [SKIP]; the summary now counts skipped checks next to passed and failed ones. Without E2E_MINIO the suite runs exactly as before.
The e2e job ran one flow on three server sets; the new suites need their own legs, and the multi-arch release image was built for arm64 only at release time, with neither pytest nor any round trip ever running on arm64 before it shipped. * e2e: one leg per suite - alerts, multijob, tls, encryption - next to the engines legs compose, lts and latest, plus mariadb-11.8, so the least-privilege MariaDB checks cover 11.4, 11.8 and 13. The suites are selected through E2E_SUITES; the existing legs keep their names and run the engines suite as before. * arm64: a leg on the GitHub-hosted ubuntu-24.04-arm runner builds the image natively (including its pytest gate) and runs the engines, alerts, multijob and encryption suites with E2E_MINIO=0 - the CS-MinIO images are amd64-only, so tls and the MinIO checks stay on amd64. * test: a matrix over amd64 and arm64; amd64 keeps the check name "🧪 Pytest", arm64 reports as "🧪 Pytest (arm64)". The job now declares contents: read instead of inheriting the workflow's write scopes. The release still waits for every test and e2e leg.
karlspace
changed the title
run_job took the alert's duration as "finished - started", but both were the run's start time (now), so every AlertEvent carried 0 s and the email channel, which shows the duration only when it is set, never showed it. The alerts e2e suite found it: no mail of the success, warning or error runs, over STARTTLS or SMTPS, had a "Duration:" line (run 37957317185, job "E2E Round Trip (alerts)"). The runner now measures the run on a monotonic clock (runner._clock), from the start of run_job to the alert, for completed and aborted runs alike. "now" keeps naming and dating the snapshot, so a caller that pins it - the tests do - still gets a real duration, and a clock step during the run cannot distort it. Regression tests: a pinned clock gives 12.5 s for a completed and 3.25 s for an aborted run, and a real run whose source takes 50 ms reports at least that; all three fail on 1fb87cf (0.0). The docs note that up to 1.10.0 no mail showed a duration.
http_post - the transport of the webhook, Teams, Slack, Discord, ntfy and Healthchecks channels - called urlopen without a timeout. A receiver that accepts the request and never answers (a hanging endpoint or proxy) blocked the run that sent the alert indefinitely, and under the daemon every later run of the job, since the scheduler starts a job only when its previous run has ended. The alerts e2e suite found it: a run posting to such a receiver was still hanging when it was killed after 180 s, and the Slack post after the webhook never arrived (run 37957317185, job "E2E Round Trip (alerts)"). urlopen now runs with HTTP_TIMEOUT_SECONDS = 30 for the connection and for each wait on the answer, the same kind of bound the email channel got for SMTP in 1.8.0. A receiver that does not answer in time fails its channel like any other delivery error - logged, the next channel still delivered. Regression tests: against a loopback socket that never answers, the post fails at a 1 s limit instead of hanging; every post passes the 30 s default to urlopen. The docs describe the bound under per-channel fault isolation.
_run_checked ran age and gpg without capture_output, so the completed
process carried no stderr and every failure read "<tool> exited with
<code>: None" - in the ERROR log line, the job error and the alert
("encryption (gpg) failed, snapshot stored UNENCRYPTED: gpg exited
with 2: None"). The tool's actual reason, e.g. "Unusable public key",
only went to the container's stderr, detached from the run. Seen in
the encryption e2e suite (run 37957317185).
The tools now run with capture_output=True, as the docstring always
assumed. The reason reaches the error; argv (recipient, paths) is
still left out of the message.
Regression tests run a real process through the engine's own
subprocess options: its stderr appears in the EncryptionError for
encrypt and decrypt. Both fail on 1fb87cf ("None"); the existing
FakeRun test could not catch it, as it returned stderr unconditionally.
restore ran "age --decrypt --output <out> <artifact>" with no
identity. age has no default identity: without -i it treats the file
as passphrase-encrypted and fails ("the file is not
passphrase-encrypted, identities are required"), so no age snapshot
could be restored by the engine, and the EncryptionError escaped the
restore command as a traceback. The encryption e2e suite found it on
the first age restore (run 37957317185).
* New optional field encryption.identity_file: the age identity file
(private key) a restore decrypts with, passed as --identity. Backups
do not use it - the backup host needs only the recipient.
* An age artifact without identity_file stops the restore before age
runs, with "cannot decrypt snapshot <id>: ... set
encryption.identity_file of job '<job>' ...". The suffix decides,
not the job's current mode, as before.
* Any decryption failure ends the restore as an error - logged, "restore
finished with errors", exit 1 - before a hook or component runs,
instead of an uncaught exception.
Compatible: identity_file defaults to null and changes nothing else;
gpg restores are unchanged. Regression tests cover the --identity argv,
a restore that passes the configured file, a missing file that fails
before age runs, and a failing age that ends without a traceback; all
fail on 1fb87cf. Docs: encryption.md (field, argv, restore, key setup)
and restore.md.
gpg --batch --encrypt refuses a recipient key whose validity the keyring cannot establish. A key that was only imported - the setup docs/encryption.md describes, and the right one for a backup host that must not hold the secret key - is such a key: gpg answered "There is no assurance this key belongs to the named user ... Unusable public key", the engine fell back to plaintext and every snapshot was stored UNENCRYPTED with a warning. The encryption e2e suite found it with a public-key-only keyring (run 37957317185). The engine now passes --trust-model always: the recipient is the operator's explicit choice in the config, so its key is used as configured. Keys that were already signed or trusted behave as before; an unknown recipient still fails and falls back as documented. Trade-off: gpg no longer asks the web of trust, which a backup container does not have. A recipient given as an email address could match a second key someone imported into the same keyring, so the docs now recommend the full fingerprint and show how to import the public key and read it. The argv test expects --trust-model always; it fails on 1fb87cf.
_produce logged a source that raised ("source <name> (<type>) failed:
<reason>"), but not one that returned its failure as an errored
component - the way the built-in sources report a failed pg_dump or
mariadb-dump, an S3 bucket they cannot reach or a missing path. The
run then ended in error with nothing in the log but "finished: error";
the reason was only in the manifest and the alert. The TLS e2e suite
found it: an S3 source refused by certificate verification logged no
CERTIFICATE_VERIFY_FAILED (run 37957317185, job "E2E Round Trip
(tls)").
Both paths now log the same ERROR line with the redacted reason. The
job errors, the manifest and the alert are unchanged.
Regression test: a filesystem source with a missing path logs
"source gone (filesystem) failed: path not found ..."; it fails on
1fb87cf. docs/sources.md mentions the log line.
The age checks end with the source file deleted (the restore without an identity must leave it missing), so the gpg backup after them captured an empty tree, and its restore check could not pass even with working encryption - one engine defect showed up as two failures in run 37957317185. The gpg section now writes the file again first.
The new suites verify behaviour the docs did not state or stated wrongly, so the docs now say what CI proves and how to set it up: * README: the suites, what each proves, how to run them, the CI legs including MariaDB 11.8 and arm64. * sources.md: postgres needs no superuser - the database owner backs up and restores (CI does, keep_acl and partitioned tables included); MariaDB gets the backup-user section MySQL already had: SELECT, SHOW VIEW, TRIGGER, EVENT and SHOW CREATE ROUTINE (11.3+), as CI grants them on 11.4, 11.8, 13 and the newest release. * restore.md: restore and verify fetch an off-site-only snapshot from the job's S3 destination themselves - the page told operators to copy it into the data volume by hand; the coverage line lists the new round trips. * notifications.md: how SMTPS trusts a private CA (SSL_CERT_FILE, and that the file replaces the CA store for the HTTP channels as well), and that CI delivers every channel to real receivers.
6ca708c gave runner._decrypt_if_needed a required job parameter, so it can read encryption.identity_file. Command plugins reuse the engine's restore front-half and call it as _decrypt_if_needed(artifact, work): CS-NocoDB, CS-n8n, CS-DocumentSigning and IP-CloudflareTerraform (code search). With the next engine release their restore commands would have failed with a TypeError for every snapshot, encrypted or not. job is optional again. Without it plain and gpg artifacts decrypt as before, and an age artifact raises EncryptionError naming encryption.identity_file - the same exception type those plugins got before, when age ran without an identity. Plugins that pass the job get age restores too; docs/plugins.md says so. Regression test: the two-argument call returns a plain artifact, raises EncryptionError for an age artifact without running age, and the call with job=... decrypts with the job's identity file. Unreleased regression; no released version is affected.
c2b5b38 moved "# nosec B310" to a line of its own above the urlopen call when it added the timeout. bandit only honours the marker on the line it reports, so the deliberate exception for the operator- configured URL was lost: bandit reports B310 on that line again. The marker is back on the call; the reason sits in the comment above it.
The MySQL healthcheck pinged over the socket ("-h localhost"). The
image's entrypoint first runs a temporary server with --skip-networking
to create the database and users, and that server answers the socket
ping, so the container turned healthy before initialization finished.
On the slower arm64 runner the engines suite then queried a server
that was restarting ("Can't connect ... through socket", run
37957317185) or had no database app yet ("Unknown database 'app'",
run 37963036742). The checks still passed only because the MySQL
section runs about a minute later.
The ping now goes over TCP to 127.0.0.1, which only the final server
listens on.
karlspace
changed the title
karlspace
marked this pull request as ready for review
October 9, 2026 17:09
Contributor
|
🎉 This PR is included in version 1.10.1 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The engine's CI round trips (
scripts/e2e.sh) covered the database, filesystem and S3-bucket sources only. The other engine features were unit-tested only, against fakes, or not at all:Change
E2E suites (
scripts/e2e.shis now a runner; each suite has its own file inscripts/e2e/;E2E_SUITESor arguments select them; the default isengines, sobash scripts/e2e.shruns what it ran before):engineskeep_acl. A MariaDB least-privilege user (SELECT, SHOW VIEW, TRIGGER, EVENT, SHOW CREATE ROUTINEon the database) backs up, and the trigger, procedure and event come back on restore; an EXECUTE-only user must fail. Events were added to the MySQL checks too.alertsmultijobprune --job, restore by a job-scoped id, SIGTERM drain, and the per-job healthcheck (a job stale by its own max age; a failure not masked by a later good run of another job).tlsca_bundleworks (includinglistof an off-site-only snapshot, hydration on restore and verify),verify_tls: falseworks and logs its warning, and the S3 source behaves the same way.encryptionCI (
docker-release.yml): one parallel leg per suite. There is a newmariadb-11.8engines leg, so the MariaDB checks cover 11.4, 11.8 and 13. The newarm64leg runs on the GitHub-hostedubuntu-24.04-armrunner: a native image build (including its pytest gate) and the engines, alerts, multijob and encryption suites withE2E_MINIO=0, because the CS-MinIO images are amd64-only.🧪 Pytestbecame an amd64/arm64 matrix; the amd64 check name is unchanged. The existing legs keep their names.Engine defects found by the new runs and fixed (each fix commit has regression tests that fail on
1fb87cf):fix(runner): reported the run's duration in alertsfinished - startedwere bothnow), so no mail ever showed a duration.fix(notify): bounded HTTP alert deliveries by a timeouturlopenhad no timeout. A receiver that never answers blocked the run, and under the daemon every later run of the job. Now 30 s.fix(encryption): surfaced the tool's error output in failurescapture_output, so every failure readgpg exited with 2: None.fix(encryption): restored age snapshots with an identity fileage --decryptwithout an identity, so no age snapshot could be restored (traceback). New opt-inencryption.identity_file. Any decrypt failure now ends the restore cleanly (exit 1) before any hook or component runs.fix(encryption): encrypted to imported gpg keys without ownertrust--trust-model always.fix(runner): logged why a source component failedfinished: error.fix(runner): kept the decrypt helper callable without a job_decrypt_if_needed(artifact, work), sojobstays optional.There are also two harness/maintenance commits: the MySQL healthcheck now pings over TCP, because the socket ping turned healthy during the entrypoint's
--skip-networkinginit (seen twice on arm64); and the# nosec B310marker is back on theurlopenline. Docs: README, sources (postgres owner, MariaDB backup user), restore (S3 hydration is automatic, which the old page did not say; decrypt failures), notifications (SSL_CERT_FILE for a private SMTPS CA, time limit, duration), encryption (identity_file, gpg trust model, error output), plugins (pass the job).How verified
Duration:in 6 mails; the stalled receiver held the run until it was killed after 180 s; Slack after it never arrived), encryption (EncryptionError: age exited with 1: Nonewith a traceback; gpgUnusable public key, UNENCRYPTED), and tls (no log reason for the failed S3 source). arm64 showed the same results.…_17-00_alpha/…_17-00_beta, SIGTERM exit 0), engines 37/37 on compose/lts/latest/11.8, arm64 89 passed + 6 skipped (MinIO checks; the image runsmariadb-dump … for Linux (aarch64)).mysql server: 26.7.0instead of an init-phase error (one run, so the race fix is shown, not statistically proven).pytest497 passed (Python 3.14.3);shellcheck -S warning,actionlintanddocker compose configare clean (no local containers were run). The alerts checker was also run against the engine's real channel classes in a local harness. bandit confirms B310 is suppressed again.Impact / compatibility
encryption.identity_file, is opt-in and defaults tonull. The fixes change only broken behaviour:identity_fileis set (before, it never could);--trust-model alwaysuses the configured recipient as given. Keys that are already trusted behave as before. A key that was only imported now encrypts instead of falling back to plaintext. The docs recommend naming the recipient by full fingerprint, so no second key in the keyring can match an email address._decrypt_if_needed(artifact, work)still works. Without the job, an age artifact raisesEncryptionError, the same type as before.fixcommits produce a patch release. A merge commit (or rebase) keeps them as separate changelog entries; a squash would collapse them into one.Follow-ups
_decrypt_if_needed(artifact, work, job), so their restore commands can decrypt age snapshots.tls: true, the default) the email channel does not verify the server certificate (smtplib default, documented), so the SMTP login goes to an unauthenticated peer. Options: opt-in verification for STARTTLS, or flip the default in a major release.option --ssl-verify-server-cert is disabled, because of an insecure passwordless loginon every dump, because the password comes fromMYSQL_PWD. DB TLS is therefore not verified. Passing it via--defaults-extra-filemight change that.publicbelongs to the bootstrap superuser).