Wire tracing diagnostics through libazureinit-kvp

[Peyton Robertson (peytonr18)]

Tracing, Diagnostics, and KVP Integration

Azure Init now uses libazureinit-kvp for diagnostics and provisioning reports instead of maintaining its own KVP encoder and writer. The reusable, synchronous DiagnosticsKvp bridge handles tracing; the application owns logging policy and report orchestration.

Architecture

Component	Responsibility
libazureinit-kvp	Pool I/O and locking, DIAG encoding/chunking, the report model and writer, CLI, and optional tracing bridge
libazureinit	Provisioning operations, Azure error-to-report mapping, and the HTTP-only wireserver module
azure-init	Subscriber setup, configuration and filters, pool cleanup, and final report delivery

Two KVP paths share the same store: diagnostics append operation history; the provisioning report replaces the final status. HTTP reporting is a separate transport.

flowchart TD
    APP["Azure Init"] --> TRACE["Tracing spans and events"]
    TRACE --> LAYER["DiagnosticsKvp Layer<br/>capture and emit"]
    LAYER --> WRITER["DiagnosticWriter<br/>validate, encode, chunk"]
    WRITER -->|append| STORE["KvpPoolStore<br/>flock + OFD fcntl"]
    APP --> RESULT["Provisioning result<br/>one typed report"]
    RESULT --> PUBLISH["publish_provisioning_report<br/>tokio::join!"]
    PUBLISH --> WRITE["write_report<br/>spawn_blocking"]
    PUBLISH --> HTTP["wireserver<br/>report_ready<br/>report_failure"]
    WRITE -->|upsert| STORE
    STORE --> POOL["Guest pool 1<br/>.kvp_pool_1"]
    POOL -. host reads later .-> HOST["hv_kvp_daemon<br/>kernel and Hyper-V host"]
    HTTP --> WS["Azure wireserver"]

Loading

Runtime Behavior

Diagnostics: on_new_span and on_close produce start/finish records with a shared operation UUID; on_record retains field updates. Events receive independent UUIDs, including events outside spans. Payloads contain the tracing target, level, and recorded fields as JSON text.

Span outcomes use an explicit diagnostic.result, otherwise infer failure from observed ERROR events and success when none were observed. Detected unwinding reports failure. This is tracing-based inference: filtering and missing instrumentation can hide errors.

Synchronous delivery: callbacks write through DiagnosticsKvp::emit() and the existing DiagnosticWriter validation and framing. No runtime, worker, queue, or drain lifecycle is required by the bridge. Callback failures go to stderr to avoid recursive tracing. Lock contention can delay callers; successful local writes do not guarantee durability or host receipt.

Final status: the binary constructs one ProvisioningReport from the actual provisioning result and attempts KVP and HTTP delivery with tokio::join!. write_report(&store, &report) runs through spawn_blocking and upserts PROVISIONING_REPORT. Failure HTTP requests reuse the report's encoding and timestamp; successful HTTP requests remain state-only.

Final reports bypass tracing filters and remain available if diagnostic-writer initialization fails after pool preparation. Both delivery results are observed without changing the provisioning exit code. Existing report-size and store-capacity limits remain in effect.

Startup and Configuration

1. main uses a scoped stderr-only bootstrap subscriber while obtaining the VM ID and loading configuration.
2. setup_layers runs through spawn_blocking, prepares the log file, and performs explicit stale-pool cleanup when KVP is enabled.
3. main installs the configured subscriber globally. Defaults are stderr at ERROR, the private 0600 log file at DEBUG, and KVP at INFO (INFO/WARN/ERROR).

AZURE_INIT_LOG controls file/console verbosity independently. KVP filter precedence is AZURE_INIT_KVP_FILTER, then telemetry.kvp_filter, then info; empty or invalid values fall through to the next source.

telemetry.kvp_diagnostics = false disables KVP setup, cleanup, diagnostics, and final-report writes without disabling local logs or wireserver HTTP. The reusable bridge itself installs no subscriber, chooses no filter, and performs no implicit cleanup; it is available behind the crate's tracing feature.

Cleanup and CI

• Remove the legacy libazureinit KVP, health, and logging modules and their unused OpenTelemetry, fs2, and sysinfo dependencies. The binary retains sysinfo for OS/kernel information.
• Update instrumentation and callers for the new diagnostics and wireserver APIs. CLI cleanup tests disable KVP to avoid touching the host pool.
• Run KVP CLI conformance checks in a separate testinit CI step using scratch pools, without launching extra provisioning runs.
• Use a private container tmpfs for the agent pool and systemd results for service completion. Collect parsed telemetry through the shipped CLI after the run, replacing the mock wireserver's POST-time snapshot. Container checks cover local pool behavior, not host retrieval.

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

❌ Patch coverage is 97.87956% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.34%. Comparing base (c6d1301) to head (2757b2a).

Files with missing lines	Patch %	Lines
src/main.rs	88.78%	25 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #319      +/-   ##
==========================================
+ Coverage   96.66%   99.34%   +2.68%     
==========================================
  Files          29       29              
  Lines        9960     9673     -287     
==========================================
- Hits         9628     9610      -18     
+ Misses        332       63     -269     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Peyton Robertson (peytonr18)]

Codecov Report

❌ Patch coverage is 95.50459% with 49 lines in your changes missing coverage. Please review. ✅ Project coverage is 98.96%. Comparing base (c6d1301) to head (47ed958).

Files with missing lines Patch % Lines
src/main.rs 83.83% 27 Missing ⚠️
src/logging.rs 86.50% 22 Missing ⚠️
Additional details and impacted files

@@            Coverage Diff             @@
##             main     #319      +/-   ##
==========================================
+ Coverage   96.66%   98.96%   +2.30%     
==========================================
  Files          29       29              
  Lines        9960     9596     -364     
==========================================
- Hits         9628     9497     -131     
+ Misses        332       99     -233     

☔ View full report in Codecov by Harness. 📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

I can circle back to this and give it another shot, but there are pieces of main.rs specifically that I don't think we can "cover" without writing strange production code. Most of the functionality in this file is covered by integration tests, but those aren't counted towards "line coverage".

I'll give updating coverage in these two files another shot, but I don't want to sacrifice better and clearer production code for the sake of hitting 100% coverage. I don't think there's value in that, though I'm open to suggestions if others disagree.

In the meantime, I'll try my best to get us closer to 100% here.