feat: proxy: expose cache size and alive counts as Prometheus metrics 1870

[joanestebanr]

🔄 Changes Summary

• The bridge tracker (proxy binary) now exposes its persistence-cache footprint and supervision counts as Prometheus metrics instead of extra fields in GET /tracker/v1/health:
  • bridgetracker_cache_size_bytes (gauge): on-disk size of the SQLite cache (main file + -wal/-shm); 0 when the in-memory backend is used
  • bridgetracker_alive_trackers (gauge): supervised bridges not yet in a terminal state
  • bridgetracker_alive_activities (gauge): from_addresses currently supervised by the activity subsystem (only registered/updated when the activity endpoint is configured)
• A small sampler goroutine refreshes the gauges every [Tracker] MetricsSampleInterval (default 15s) (the repo's prometheus helpers are push-style: GaugeSet)
• The proxy binary gains a [Prometheus] section (Enabled/Host/Port), with the same defaults as aggkit's (enabled, localhost:9091); startPrometheusHTTPServer moved from cmd to a new shared metrics package (metrics.StartPrometheusHTTPServer) used by both binaries
• GET /tracker/v1/health is left untouched: no new fields and HealthResponse.APIRevision is not bumped

⚠️ Breaking Changes

• None for the API: /health is unchanged. Behavior note: since [Prometheus] is enabled by default (like aggkit), the proxy now opens a metrics listener on localhost:9091 unless Enabled = false

📋 Config Updates

• 🧾 Diff/Config snippet: new [Prometheus] section in the proxy's config (proxy/config/default.go), same fields and defaults as aggkit's. Set Enabled = false to not open the metrics port; new [Tracker] MetricsSampleInterval sets the gauges' refresh period (<= 0 falls back to 15s); override Host (e.g. 0.0.0.0) if Prometheus scrapes from outside the host/container

+[Prometheus]
+Enabled = true
+Host = "localhost"
+Port = 9091
+
+[Tracker]
+MetricsSampleInterval = "15s" # how often the gauges are refreshed

🔌 API Updates

• None. No REST API change; the new surface is the Prometheus /metrics endpoint (unless [Prometheus] Enabled = false)

Example (default config)

curl -s http://localhost:9091/metrics | grep ^bridgetracker_
# bridgetracker_cache_size_bytes 4.718592e+06
# bridgetracker_alive_trackers 12
# bridgetracker_alive_activities 3

✅ Testing

• 🤖 Automatic: unit tests for the sampler (memory vs. disk, CacheStats/count errors keep the previous value, activity disabled) and for CacheStats against real SQLite files (bridgetracker/db/sqlite_registry_test.go)
• 🖱️ Manual: make lint, go test ./bridgetracker/... ./proxy/...; scrape /metrics on a running proxy with a disk-backed and an in-memory tracker

🐞 Issues

• Closes feat: proxy: expose cache size and alive trackers/activities as Prometheus metrics #1870

📝 Notes

• Why metrics instead of /health: size and counts are time series meant for dashboards/alerting, not liveness data; /health stays cheap and side-effect free
• The tracker runs in the proxy binary, which had no Prometheus integration before, hence the new config section
• Cache size reuses domain.CacheStatsProvider (optional capability, mirroring Triggerable) already implemented by the SQLite adapters. When registry and activity store share one SQLite file, the size is read once from the registry
• Docs: new "Prometheus Metrics" section in docs/bridgetracker.md

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-25T11:27:15.457990Z	e76f34e	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e76f34ee33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Include the WAL in reported disk usage

NewSQLiteDB enables WAL mode in db/sqlite.go, but page_count * page_size measures the database's logical page space rather than the current footprint of the database plus its -wal sidecar. During normal or update-heavy operation, the WAL can remain allocated or grow while page_count stays unchanged, so /health can substantially underreport the cache's actual disk usage. Retain the database path and include the WAL file size, or describe this field as logical database size instead.

Useful? React with 👍 / 👎.

[joanestebanr]

Obsolete after moving to Prometheus: CacheStats now sums the main DB file plus its -wal/-shm sidecars (sqliteFileSize) and is exposed as the bridgetracker_cache_size_bytes gauge; /health is unchanged. Verified against a live instance: gauge == sum of the three files.

[chatgpt-codex-connector]

Avoid reporting zero when counting trackers fails

When a SQLite query fails—for example because the database is temporarily unavailable—this branch leaves the non-optional AliveTrackers field at its zero value and still returns 200. A consumer therefore cannot distinguish “no active trackers” from “the count could not be read,” potentially producing incorrect operational decisions. Make the field nullable/omittable on failure or expose an explicit unavailable/error state.

Useful? React with 👍 / 👎.

[joanestebanr]

Obsolete after moving to Prometheus: /health no longer carries alive_trackers. In the sampler a failed count logs a warning and the gauge keeps its previous value instead of reporting 0.

[joanestebanr]

@claude review

[claude]

Claude finished @joanestebanr's task in 56s —— View job

---

Review of PR #1871: bridgetracker cache size and alive counts as Prometheus metrics

I reviewed the full diff against origin/develop by reading the code. I could not run go build, go test or make lint here because those commands need approval in this environment, so please rely on CI for those.

🚀 What's New

• The bridge tracker (the proxy binary) now exposes three Prometheus gauges: bridgetracker_cache_size_bytes, bridgetracker_alive_trackers and bridgetracker_alive_activities. The last one is only registered when the activity endpoint is configured.
• A sampler goroutine (BridgeTracker.StartMetricsSampler) refreshes the gauges every [Tracker] MetricsSampleInterval (default 15s).
• The proxy binary gains a [Prometheus] section (enabled by default, localhost:9091). startPrometheusHTTPServer moved from cmd into a new shared metrics.StartPrometheusHTTPServer, used by both binaries.
• CacheStatsProvider is a new optional capability on the stores, implemented by the SQLite adapters. It sums the main DB file and its -wal/-shm sidecars, which addresses the earlier Codex WAL comment.
• GET /tracker/v1/health is unchanged. This also makes the earlier Codex /health comment (zero when counting fails) moot, because the gauges keep their previous value on error.

🐛 Bug Fixes

None. The shared server also switches err == http.ErrServerClosed to errors.Is, which is a small improvement.

📋 Config Updates

[Prometheus]
Enabled = true          # set false to not open the metrics port
Host = "localhost"
Port = 9091

[Tracker]
MetricsSampleInterval = "15s"   # <= 0 falls back to 15s

⚠️ Breaking Changes

None for the API. Because [Prometheus] is enabled by default, the proxy now listens on localhost:9091 unless Enabled = false.

---

Findings

🟡 Suggested

1. alive_trackers is computed by loading and decoding every active row. metricsSampler.sample() (bridgetracker/metrics_sampler.go:70) calls GetTrackerActives(nil) and takes len(...). In the SQLite adapter that is SELECT * FROM tracked_bridge WHERE terminal_since = 0, then a JSON decode of each row (bridgetracker/db/sqlite_registry.go:663). It runs every 15s only to get a count, and the cost grows with MaxTrackedBridges. The same method is called on every engine poll tick, so this adds contention on the same SQLite file.

   • A SELECT COUNT(*) ... WHERE terminal_since = 0 would be cheaper.
   • It would also count undecodable or stale rows, which GetTrackerActives skips. Decide which behaviour you want for the metric.
   • If you keep GetTrackerActives, at least note the cost in the sampler doc.

2. sqliteActivityStore.CacheStats is never used in production. The sampler only type-asserts b.supervised (metrics_sampler.go:60). The activity store's CacheStats (sqlite_activity_store.go:741), its new dbPath field and its test are therefore dead code outside tests. The PR description says the size is "read once from the registry", which matches the wiring. The risk is that the proxy would report only the registry's file if the activity store were ever opened on a different path. Either remove the activity-store implementation (YAGNI), or make the sampler sum distinct paths.

3. Stale and misleading comments.

   • bridgetracker/domain/ports.go:147-149: the CacheStats doc still says it is "computed via PRAGMA page_count * page_size — the adapters do not otherwise retain the file path". The implementation now uses os.Stat on the retained dbPath plus the WAL/SHM sidecars.
   • proxy/config/config.go:59: the comment says "Prometheus configures the metrics server (disabled by default)", but it is enabled by default.
   • bridgetracker/db/sqlite_activity_store.go:737: the CacheStats doc says "published by the Prometheus sampler", but the sampler never calls it on the activity store (see item 2).

4. Port clash between aggkit and proxy. Both binaries now default to localhost:9091. If an operator runs them on the same host, whichever starts second fails in net.Listen. StartPrometheusHTTPServer only logs the error and returns, so the process keeps running with no metrics. This is not new behaviour for the helper, but the proxy now inherits it by default. Consider making a listen failure in the proxy more visible, or at least mention the clash in docs/bridgetracker.md.

5. The metrics server has no shutdown path. StartPrometheusHTTPServer creates its http.Server internally and never shuts it down. Aggkit's helper had the same limitation, so this is not a regression. Since the helper is being moved into a shared package, it could accept a ctx or return the server so the proxy can shut it down on exit.

6. Unrelated generated file. bridgetracker/mocks/mock_bridge_address_resolver.go (+144) has nothing to do with this feature. No test uses it, and bridgetracker/bridge_address_test.go has its own hand-rolled fake. It looks like a make generate-mocks side effect. Consider dropping it from this PR to keep the diff focused.

🟢 Optional

7. sample() could be split. metrics_sampler.go:59 mixes three independent probes with nested if/else. A small helper per gauge would read more cleanly and avoid the nested else blocks.

8. Test isolation. The sampler tests use the package-global prometheus state (prometheus.Init() is a sync.Once and the gauges map is shared). They pass in order today, but TestMetricsSamplerMemoryBackedReportsZeroSize registers with withActivity=false while alive_activities is already registered by an earlier test. If you ever assert that the gauge is absent when activity is off, that test would give a false result. Not a problem now.

9. WAL/SHM test uses a fake -wal file. TestSQLiteRegistryCacheStatsIncludesWALSidecar writes a fake -wal file next to a live DB to prove the sidecar is counted. That works for the unit test, but nothing checks real WAL growth. It is optional.

🔴 Required

None found.

Security and performance notes

• Binding to localhost by default is the safe choice. The docs correctly point out that Host must be overridden for scrapes from outside a container.
• The metrics do not contain addresses, hashes or other user data. The gauges carry no labels, so there is no cardinality risk.
• os.Stat on three paths every 15s is negligible. Item 1 is the only meaningful performance concern.
• dbPath is used verbatim in os.Stat. NewSQLiteDB builds a file:%s?... URI from it, so a dbPath containing ? or a file: prefix would stat a different path than SQLite opens. This is an edge case, and Config.DBPath is operator-controlled.

Overall

The design is sound and the Codex WAL comment is properly addressed. Items 1 to 3 are the ones I would fix before merge, and item 6 is easy cleanup. CI should confirm lint and tests.

[joanestebanr]

Thanks for the review. Addressed in 39266b0:

• Improve error tracing #6 unrelated mock: my mistake, a stray untracked mock_bridge_address_resolver.go got committed. Removed from the PR.
• Add UTs for lastgersync #3 stale comments: fixed domain/ports.go (CacheStatsProvider/CacheStats docs), proxy/config/config.go ("enabled by default") and the sqliteFileSize doc.
• Split L1 / L2 responsibilities for lastgersync #2 activity-store CacheStats: removed (with its dbPath field and test). Only the registry is sampled; the activity store opens the same file, which CacheStatsProvider's doc now says.
• Dev Docs #4 port clash: documented in docs/bridgetracker.md (aggkit and the proxy both default to localhost:9091; the second to start logs the bind error and runs without metrics).

Not changed, on purpose:

• Unify L1 syncers #1 alive_trackers via GetTrackerActives: agreed it is heavier than a COUNT(*), but a count would also include rows GetTrackerActives skips (stale/undecodable) and needs a new method on the SupervisedStore port plus regenerated mocks. Cost is bounded by MaxTrackedBridges every 15s (configurable via MetricsSampleInterval). Happy to do it as a follow-up if it shows up as contention.
• Add monitoring #5 server shutdown: same limitation as aggkit's helper before the move; out of scope here.
• Update target contract for aggOracle #7-Add E2E test for aggOracle using kurtosis env #9: optional, left as is.