perf(boot): wait for readiness on the host and take the container id from docker run

[mhenrixon]

Summary

Two round trips come out of every boot, and one of them is the row that scaled with how long a container takes to come up.

The readiness wait moves onto the host. Dash::Commands::App#wait_for_ready builds a shell loop that evaluates the same status Dash::Cli::Healthcheck::Poller used to read one round trip at a time — docker inspect's health status, or the healthcheck: exec: probe's exit code — returns the moment that status is one the poller accepts, and otherwise keeps looking until deploy_timeout. Dash::Cli::App::Boot runs it as a single capture with an interaction handler attached, so the Container not ready yet, retrying in 1s (Xs elapsed, Ys left) beacons still print once a second while it waits. Progress goes to stderr and the final status to stdout, which is what keeps the captured value the status and nothing else.

Reaching the deadline exits 0 — it is an answer, and the poller phrases it. A non-zero exit means only that the status could not be read at all (docker unreachable, container gone), which is a broken command and SSHKit's to raise, exactly as it was when that read was a round trip of its own.

The proxy target comes out of docker run. docker run --detach prints the id of the container it just started; boot captures it and takes the first twelve characters — the same short id docker container ls --quiet printed — instead of asking docker for it again on the next round trip.

Every readiness decision still lives in the poller, word for word: drift, announce_missing_gate, the readiness-delay confirm, the timeout error. The host loop only decides when to return, never what it means — Dash::Commands::Base::READY_STATUSES is the one thing both sides read, so they cannot drift apart.

Closes #163

Round trips per host, per role shape

Counted from the boot path and pinned by tests (test/cli/app_test.rb), not from a live deploy — see Deviations.

Round trip	web (proxy)	job (healthcheck, no proxy)	unchecked (no healthcheck)
boot_state	1	1	1
audit + ensure_env_directory (upload! is SFTP)	1	1	1
docker run --detach	1	1	1
container_id_for_version	1 0	–	–
dash-proxy deploy	1	–	–
readiness wait	–	N 1	N 2
stop old version	1	1	1
clean_up_assets	1	–	–
total	7 → 6	4 + N → 5	4 + N → 6

On the 4-host deploy in the issue (N ≈ 5 on the job host) that is web 7 → 6 and job 9 → 5.

• The container_id_for_version saving is a skip — the answer was already on the host's stdout.
• The readiness saving is a fold: N attempts become one blocking command. It is the row that matters, because N grows with boot time — a container that takes 60s cost ten round trips, and now costs one.
• The unchecked role keeps a second round trip: its readiness delay is spent on the laptop, so the confirming read afterwards has to be its own command.

unhealthy still waits — the issue's one reversed decision

The issue proposed returning from the wait the moment docker reports unhealthy, and invited the executor to keep the retry if there was a reason. There is one, and it is decisive: dash's default healthcheck emits --health-interval 1s with no --health-start-period, and docker's default --health-retries is 3. A container whose app needs longer than ~3s to serve /up is therefore reported unhealthy at t≈3s and healthy when it finishes booting. Today's poller waits through that and accepts the boot; returning early on unhealthy would fail essentially every normal Rails boot at three seconds.

So the host loop returns early for exactly the statuses Poller#acceptable? accepts and waits out the deadline for everything else — which is precisely what the client-side poll did, attempt by attempt. The only behaviour that changes is the beacon cadence: a fixed second instead of 1s, 2s, 3s … , which is strictly more responsive.

Test plan

• bundle exec ruby -Itest -e 'Dir["test/**/*_test.rb"].grep_v(/integration/).each { |f| require File.expand_path(f) }' — 1899 runs, 0 failures
• bundle exec rubocop --parallel — no offenses
• bin/test — full suite, 1918 runs, Docker + ghcr.io/zoolutions/dash-proxy:v1.1.0.1 (MINIMUM_VERSION unchanged). The integration deploys run the wait loop on real Docker-in-Docker hosts, including app_with_roles' unchecked workers role
• docs/ — bundle exec rspec spec/config_docs_spec.rb green after the doc-wording updates
• The generated loop run under a real POSIX sh against a fake docker: starting → healthy exits 0 with the status on stdout and beacons on stderr; never-ready exits 1 at the deadline with the last status; no-healthcheck:running returns immediately; timeout: 0 makes exactly one observation without spinning; the exec branch loops on the probe's exit code
• New unit coverage: exact command strings for both branches (test/commands/app_test.rb), the streaming handler against chunks split mid-line and against an interleaved stdout status (test/cli/healthcheck/progress_reporter_test.rb), the poller's new call pattern (test/cli/healthcheck/poller_test.rb), and at the boot level — the proxy target read from the run, one wait round trip for a healthchecked role, wait-plus-confirm for an unchecked one, the streaming handler wiring, a deadline that fails once rather than waiting twice, and a boot whose status read fails reporting that instead of waiting out the timeout
• A real multi-host deploy against a staging target (before/after per-host rows) — not run in this session, no hosts available; the integration harness covers the shell loop end to end but reports no timings
• Ctrl-C during the wait leaves no orphaned loop on the host — not verified

Deviations & judgment calls

Deviation — unhealthy does not return early from the host loop. Reasoned above; it is the one decision in the issue's Decision section I reversed, and the tests pin the behaviour either way.

Judgment — the block protocol is |mode, seconds_left|. The poller needs two different reads from the CLI: the blocking wait, and the plain status confirm after a readiness delay. Splitting wait_for_healthy into two callables would have churned every poller test, so the block is called with :wait / :confirm instead; procs ignore extra args, so the existing tests' blocks were untouched. seconds_left is passed so a retried wait cannot overshoot deploy_timeout.

Judgment — the run capture is unconditional, not proxy-only. One code path reads better than branching execute/capture on running_proxy?, and the id is what the failure message is about for every role.

Judgment — stub_capture echoes the captured command into SSHKit's output. A stubbed capture_with_info is intercepted above the Printer and never printed, so moving docker run from execute to capture made it invisible to roughly a dozen assertions across three test files. Rather than rewrite them all to read a recorded array, the shared stub helper echoes what it answered into the same stream stdouted reads. Side effects in a mocha matcher are not lovely; recorded_commands and stub_boot_state already use the idiom.

Discovery — the docs described the old cost model. lib/dash/configuration/docs/role.yml and docs/app/views/docs/pages/worker_roles.rb both claimed an exec probe costs "an SSH round trip plus a process spawn per poll" and that dash "polls docker's verdict with backoff". Neither survives this change; both updated.

Discovery — a latent bug next door, left alone. Dash::Cli::Main#container_available? rescues SSHKit::Runner::MultipleExecuteError, which sshkit 1.25 does not define, so a rollback to a version whose container is missing raises NameError instead of the intended message. Surfaced when a test stub moved; out of this issue's path and not touched here. Worth its own issue.

Review round (94be4cb)

Three findings from the cubic pass, two of them regressions this PR had introduced. All resolved.

1. The wait swallowed a broken status read. 2>/dev/null plus an ignored exit status turned an unreachable daemon or a vanished container into an empty status the loop waited out for the whole deploy_timeout, then blamed the container. Before this PR that read raised on the spot with docker's own error. Restored: the inspect read no longer redirects stderr and ends in || exit $?, the CLI no longer passes raise_on_non_zero_exit: false, and the deadline exits 0 instead — so a non-zero exit means exactly one thing. The exec branch is untouched, because there a non-zero exit is the answer "not ready".
2. The progress reporter buffered both streams into one buffer. stdout and stderr are separate SSH streams whose chunks can interleave, so the final status could land inside a half-arrived progress line. It now reads stderr only — the stream is part of the wait's contract, not something the line regex should be left to infer.
3. Two stub_readiness_confirm calls asserted nothing because they never ran. deploy_with_accessories.yml and deploy_with_proxy.yml both set readiness_delay: 0, and the poller only makes the confirming read when the delay is non-zero. The dead stubs are gone; the four call sites where the confirm genuinely happens now assert it with expect: true.

Re-verified under a real POSIX sh with GNU xargs semantics (what Linux deploy hosts run): container gone → exit 123 with Error: No such object; daemon down → exit 123 with Cannot connect to the Docker daemon; deadline → exit 0 with the last status; healthy → exit 0 at once. bin/test green at 1921 runs.

Follow-up (1774d57). The re-review flagged that || exit $? alone is not a portable signal, and it was right about the mechanism even though its example platform is not one dash deploys to. The read is a pipeline, so its exit code is xargs': GNU (no -r) runs docker inspect with no container when nothing is piped and exits 123, while BSD and BusyBox skip the utility and exit 0. Both leave the status empty, and empty is not something a working docker inspect --format can print — so that is checked too, and the failure path no longer depends on the host's findutils.

Verified by running the generated command under both: debian:stable-slim (GNU findutils 4.10.0, /bin/sh → dash) exits 123 with Cannot connect to the Docker daemon on stderr, macOS/BSD exits 1 with it, and starting → deadline exits 0 with the last status while healthy exits 0 at once on both.

[cubic-dev-ai]

All reported issues were addressed across 14 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 8 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}