feat(healthcheck): add --health-start-interval for run/create by akashchamp · Pull Request #5229 · containerd/nerdctl

akashchamp · 2026-09-23T22:15:56Z

Summary

Implements --health-start-interval for nerdctl run/nerdctl create, closing the Docker CLI compat gap tracked in #3867 (still listed there as unimplemented as of the most recent audit on 2026-08-29).

Docker's --health-start-interval lets a container probe more frequently while it is still inside --health-start-period, then fall back to the regular --health-interval cadence once the start period has elapsed. nerdctl already accepted --health-start-period but had no way to probe faster during it, and docs/command-reference.md/docs/healthchecks.md explicitly called out --health-start-interval as unimplemented.

Adds the --health-start-interval flag to run/create, mirroring the existing --health-start-period plumbing: ContainerCreateOptions, flag validation (rejects negative values), the Healthcheck struct that's stored on the container label and read back via inspect, and defaults to Docker's 5s when unset (ApplyDefaults).
Threads it through nerdctl compose via healthcheck.start_interval (was previously silently ignored with a TODO, and listed as unimplemented in docs/compose.md).
The one real logic change: probe scheduling. nerdctl drives probes with a single systemd timer created once per container (CreateTimer); that timer's own cadence can't safely be changed at runtime while the unit it triggers is still active — and it is active for the whole duration of the very nerdctl container healthcheck <id> invocation it just triggered. So instead the timer now ticks at the faster of --health-interval and --health-start-interval, and ExecuteHealthCheck skips any tick that arrives before the interval that actually applies right now (--health-start-interval while inside the start period, --health-interval once it's elapsed or a healthy result has ended it early) has elapsed since the last probe that actually ran. This keeps the systemd unit untouched for the container's whole lifetime while still producing the Docker-compatible effective probe cadence.
Updates docs/command-reference.md, docs/healthchecks.md, and docs/compose.md to reflect the flag as implemented.

Test plan

go build ./..., go vet ./... on the touched packages, and cross-compiles for windows/freebsd/darwin (the scheduling logic lives in OS-shared pkg/healthcheck/executor.go, so it has to build everywhere even though only Linux actually runs it).
go test ./pkg/healthcheck/... ./pkg/composer/... ./pkg/inspecttypes/... — added pkg/healthcheck/executor_test.go (table-driven, pure-function tests for the new scheduling decision) and extended pkg/composer/serviceparser/serviceparser_test.go for the compose field.
Extended cmd/nerdctl/container/container_run_test.go (TestRunHealthcheckFlags, incl. a negative-value validation case) and cmd/nerdctl/container/container_health_check_linux_test.go (TestContainerHealthCheckDefaults for the new default/override, plus two new subtests under TestContainerHealthCheckAdvance that manually drive nerdctl container healthcheck to assert the start-interval cadence during the start period and the fallback to health-interval afterward).
Manual verification against a real container on rootful containerd + systemd (not just unit tests): ran nerdctl run -d --health-cmd "exit 1" --health-interval 20s --health-start-period 6s --health-start-interval 1s ... and watched nerdctl inspect ... --format '{{json .State.Health}}' over time. During the 6s start period, probes landed roughly every 1.3-1.8s (four of them, --health-start-interval=1s plus systemd's 1s AccuracySec jitter). After the start period ended, the next probe landed ~21.7s after the last one — matching --health-interval=20s, not the 1s start-interval. systemctl show <id>.timer -p TimersMonotonic confirmed OnUnitInactiveUSec stayed fixed at 1s throughout, i.e. the cadence switch is entirely the new software-side throttle in ExecuteHealthCheck, not a timer reconfiguration. Cleaned up the test container, timer, and pulled image afterward.
golangci-lint was not run locally (not installed on the build host); relying on CI for that pass, go vet was clean.

Notes

Competing-work / saturation check before starting: no open PR, fork, or branch implements this flag; the only prior PR referencing it (docs: update unimplemented Docker features list #5183, closed) only documented it as unimplemented and didn't touch the actual behavior.
Part of this PR (implementation, tests, and this description) was drafted with AI assistance (Claude); I reviewed, tested (including the manual verification above), and understand the change described here.

Docker's `--health-start-interval` lets a container probe more frequently while it is still inside `--health-start-period`, then fall back to the regular `--health-interval` cadence once the start period has elapsed. nerdctl accepted `--health-start-period` but had no way to probe faster during it, and docs/command-reference.md still listed `--health-start-interval` as unimplemented. This wires the flag through run/create exactly like the existing `--health-start-period` plumbing (CLI flag, ContainerCreateOptions, validation, the Healthcheck struct that gets stored in the container label and read back on inspect), defaults it to Docker's 5s when unset, and threads it through `nerdctl compose` via `healthcheck.start_interval`. The one real behavioral change is probe scheduling. nerdctl schedules probes with a single systemd timer created once per container, whose tick cadence can't safely be changed while the very unit it triggers is still running (that unit is what invokes `nerdctl container healthcheck`). So the timer now ticks at the faster of `--health-interval` and `--health-start-interval`, and `ExecuteHealthCheck` skips ticks that arrive before the interval that actually applies (start-interval while in the start period, interval afterward) has elapsed since the last probe that really ran. This keeps the systemd side untouched for the container's lifetime while still producing the Docker-compatible effective cadence. Verified with a real container against rootful containerd + systemd: with --health-interval=20s --health-start-period=6s --health-start-interval=1s, probes ran roughly every ~1.3-1.8s during the start period, then the next probe landed ~21.7s after the last one (matching --health-interval, not --health-start-interval) once the start period ended - while `systemctl show` confirmed the timer's OnUnitInactiveUSec stayed fixed at 1s throughout, i.e. the cadence switch is purely the new software-side throttle, not a timer edit. Signed-off-by: Akash Kumar <116457960+akashchamp@users.noreply.github.com>

haytok

CI fails:

https://github.com/containerd/nerdctl/actions/runs/35927358984/job/107406988024?pr=5229

=== Failing tests ===
TestContainerHealthCheckAdvance
TestContainerHealthCheckAdvance/Health_check_probes_at_start-interval_cadence_within_the_start_period
TestContainerHealthCheckAdvance/Health_status_transitions_from_healthy_to_unhealthy_after_retries
=====================

Also, linting fails:

https://github.com/containerd/nerdctl/actions/runs/35927358875/job/107405512522?pr=5229

akashchamp · 2026-09-24T20:30:49Z

Fixed both: split TestContainerHealthCheckAdvance under the function-length lint limit into its own TestContainerHealthCheckStartInterval, added --health-interval 1s to the retries subtest so its manual probes aren't throttled by the 30s default, and loosened the start-interval subtest's exact len(h.Log) == 2 assertion to >= 2 since the container's own background ticker can legitimately add an extra probe in that window.

…versized test func Three CI failures on this branch, all confined to tests this PR itself touches or added: - "Health status transitions from healthy to unhealthy after retries" (pre-existing test) now fails because probes are throttled to run no more than once per --health-interval (see shouldRunProbe). That test doesn't set --health-interval, so the 30s default applied while the test drives 4 manual "container healthcheck" calls only 2s apart - only the first one actually ran, so FailingStreak never reached the configured 3 retries. Pass --health-interval 1s explicitly so each manual call is spaced beyond the throttle window, matching the pre-existing intent of the test. - "Health check probes at start-interval cadence within the start period" (new in this PR) asserted exactly 2 log entries after 2 manual "container healthcheck" invocations. With --health-start-interval 1s, the container's own background systemd timer also ticks every 1s during the start period and can legitimately land a probe in the ~2s window between the two manual calls, producing a 3rd log entry. Relax the assertion to a lower bound (>= 2) with a comment explaining why, since the property under test - that manual ticks aren't skipped - doesn't require an exact count. - golangci-lint (revive function-length, max 500 lines) failed because the two new start-interval subtests pushed TestContainerHealthCheckAdvance to 553 lines. Move them into a new TestContainerHealthCheckStartInterval function (479 and 103 lines respectively after the split). The EL/almalinux-8 rootful/rootless failures (TestComposeCopy, TestCopyToContainer, TestCopyFromContainer) come from workflow-flaky.yml, this repo's own known-flaky/experimental job bucket, and don't touch any file this PR changes. The gomodjail rootless (TestRunRmTime, a hard wall-clock deadline) and rootless-port-slirp4netns (TestLoadQuiet) failures are likewise in files untouched by this diff. All three are left alone as external. Verified: go build ./..., go vet ./pkg/healthcheck/... ./cmd/nerdctl/container/..., go test ./pkg/healthcheck/..., gofmt/goimports, and golangci-lint on the touched packages; also confirmed by direct line count that both split functions are under the function-length limit. Signed-off-by: Akash Kumar <116457960+akashchamp@users.noreply.github.com>

haytok

The code comments are too long (since they may be generated by AI). Could you shorten them in your own words so they stay readable and easy to maintain?

Also, could you add a case for HEALTHCHECK --start-interval from the Dockerfile in TestRunHealthcheckFromImage?

haytok · 2026-09-26T13:16:58Z

+	if hc.StartPeriod > 0 && hc.StartInterval > 0 && hc.StartInterval < tickInterval {
+		tickInterval = hc.StartInterval
+	}


nit:

Suggested change

if hc.StartPeriod > 0 && hc.StartInterval > 0 && hc.StartInterval < tickInterval {

tickInterval = hc.StartInterval

}

if hc.StartPeriod > 0 && hc.StartInterval > 0 {

tickInterval = min(tickInterval, hc.StartInterval)

}

haytok · 2026-09-26T14:23:31Z

+func shouldRunProbeAt(now, containerCreated, lastProbeAt time.Time, inStartPeriod bool, hc *Healthcheck) bool {
+	// No prior probe recorded: always run the first one.
+	if lastProbeAt.IsZero() {
+		return true
+	}
+
+	target := hc.Interval
+	if hc.StartPeriod > 0 && inStartPeriod && now.Sub(containerCreated) < hc.StartPeriod {
+		target = hc.StartInterval
+	}
+	return now.Sub(lastProbeAt) >= target
+}


Based on healthcheck interval on Docker, if the time left in the start period (since the last probe) is shorter than start-interval, the next probe should be due after that remaining time instead.

https://github.com/moby/moby/blob/master/daemon/health.go#L265C2-L285C3

haytok · 2026-09-26T17:17:27Z

+					// Probes are now throttled to run no more often than --health-interval
+					// (see shouldRunProbe), which otherwise defaults to 30s: shorter than
+					// this test's manual invocations are apart, so each of them would
+					// otherwise be skipped as "not due yet" instead of actually probing.


When the throttle skips a manual nerdctl container healthcheck, the command exits 0 and prints nothing, so it looks exactly like a successful probe, which is confusing. We should at least log the skip at Info or Warn level.

AkihiroSuda · 2026-10-02T08:31:05Z

Needs rebasing

haytok suggested changes Sep 24, 2026

View reviewed changes

addemod mentioned this pull request Sep 25, 2026

fix(compose): support cgroup_parent and cgroup service fields #5230

Merged

akashchamp force-pushed the health-start-interval branch from 29f6a38 to 396e3ac Compare September 25, 2026 18:24

haytok suggested changes Sep 26, 2026

View reviewed changes

Navigation Menu

feat(healthcheck): add --health-start-interval for run/create - #5229

feat(healthcheck): add --health-start-interval for run/create#5229
akashchamp wants to merge 2 commits into
containerd:mainfrom
akashchamp:health-start-interval

akashchamp commented Sep 23, 2026 •

edited

Loading

Uh oh!

haytok left a comment

Uh oh!

akashchamp commented Sep 24, 2026

Uh oh!

haytok left a comment

Uh oh!

haytok Sep 26, 2026

Uh oh!

haytok Sep 26, 2026

Uh oh!

haytok Sep 26, 2026

Uh oh!

AkihiroSuda commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

akashchamp commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Notes

Uh oh!

haytok left a comment

Choose a reason for hiding this comment

Uh oh!

akashchamp commented Sep 24, 2026

Uh oh!

haytok left a comment

Choose a reason for hiding this comment

Uh oh!

haytok Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

haytok Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

haytok Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

akashchamp commented Sep 23, 2026 •

edited

Loading