buildkitd: add on-demand release of unreferenced cache metadata

[Yeaury]

Related to #7175.

Motivation

BuildKit releases unreferenced solver cache metadata during cache manager initialization, after background GC reclaims space through a throttled callback, and after an explicit prune removes records. Operators do not currently have a dedicated endpoint to request this metadata cleanup independently of pruning worker cache.

This adds an on-demand maintenance trigger. It does not solve the underlying cost of ReleaseUnreferenced or move automatic cleanup into maintenance windows.

Changes

• Add POST /debug/cache/release-unreferenced on the existing debug listener. The request runs synchronously, returns 204 No Content on success or 500 Internal Server Error on a release error, and logs elapsed time.
• Serialize controller-triggered release passes from background GC, explicit prune, and the debug endpoint with a context-aware semaphore. Queued callers can cancel; otherwise each request runs its own pass.
• Wait for active release work before closing controller resources and reject release calls after shutdown.
• Check cancellation during solver metadata traversal and after result lookups, so a canceled lookup is not interpreted as a missing result. Propagate storage release errors, except ErrNotFound for an already-removed result.
• Use the original prune stream context for deferred cleanup: the errgroup context is canceled by Wait even on success.
• Register the debug listener after controller initialization to supply the callback. Profiling and other debug endpoints consequently become available later during daemon startup.
• Document the endpoint and add handler, serialization, cancellation, closed-controller, and storage-error tests.

Usage and scope

With the debug listener bound to 127.0.0.1:6060:

curl --fail-with-body --request POST \
  http://127.0.0.1:6060/debug/cache/release-unreferenced

The debug listener should only be accessible to trusted operators. Automatic cleanup and its existing throttle remain enabled. This endpoint does not prune snapshots or blobs, compact cache.db, shorten bbolt transactions, or require taking buildkitd offline. Cancellation is checked while waiting and between storage operations; an in-flight storage operation is not interrupted. Initialization cleanup runs before the controller is available.

Validation

Passed locally:

• go test -race -count=1 ./cmd/buildkitd ./control ./solver/bboltcachestorage
• go test -race -count=1 ./solver -run '^TestReleaseUnreferenced'
• go test -count=1 -skip Integration ./solver
• go vet ./cmd/buildkitd ./control
• git diff --check

Broader validation did not pass:

• The solver race suite reports a race in TestStaleEdgeMerge, between test comparison and progress updates. The same failure reproduces with the unmodified base version of solver/cachemanager.go supplied via a Go overlay.
• Including ./solver in go vet reports an existing discarded cancellation function at solver/jobs.go:719, which this change does not modify.
• make validate-all failed in the full integration test run. Those integration failures have not been triaged; this PR does not claim full validation passes.