feat: measure real GCP usage in deployml costs

[andrewtclim]

Supersedes #64 (its commits da56a5f and ed65724 are carried over here, so nothing is lost). Closing that one.

deployml estimate got usable in #64: it feeds infracost a light/heavy usage profile and splits the output into always-on vs usage-based. But deployml costs never got the same treatment — it just re-ran the zero-usage scan against the deployed Terraform, so it printed exactly the misleading numbers the redesign existed to kill: Cloud Run, BigQuery and GCS at $0, only Cloud SQL real. It answered "what would this cost if nobody used it", which isn't a question anyone has. The follow-up section of #64 named this as the next step.

This finishes the pair: estimate = before you deploy, costs = what you're actually running at.

summary

• deployml costs now measures real usage instead of assuming it. New src/deployml/utils/measured_usage.py reads Cloud Monitoring over a window (--days, default 30), maps each metric to the infracost usage key it corresponds to, and keys it to the Terraform address. Why: once the stack exists we don't have to guess, and the student's actual behaviour is the only number that answers "what is this costing me".
• Usage is pinned per resource, not per type. render_usage_yaml() grew a resource_usage: block keyed by Terraform address. Why: infracost's resource_type_default_usage applies one number to every resource of a type, so 50k requests measured across three Cloud Run services would bill 3×50k.
• Measured usage retires the guessed profiles. New classify_usage_profile() compares real request volume against light/heavy on a log scale. Why: light/heavy stop being pricing inputs the student is billed for and become a label — "you are a LIGHT user".
• Nothing is invented. The scan gets an empty profile plus measured per-address usage only, so anything Monitoring couldn't report stays at zero and the reason prints under "About these numbers". Why: a silent $0 is the bug being fixed; an explained $0 is fine. The three approximations are disclosed in the same section (GCS class A/B inferred from method names; project-scoped BigQuery bytes split evenly across tables; Cloud Run concurrency left at infracost's default).
• No new dependency. requests was already a dep, and the Monitoring REST API is called with an ADC token from gcloud auth application-default print-access-token. Why: matches the repo's run_tool shell-out style rather than pulling in google-cloud-monitoring.
• deploy's confirmation prompt no longer shows a number we know is wrong. It used the zero-usage scan; it now uses the same usage-aware estimate. Why: it was asking students to approve a misleading figure. This also let run_infracost_scan, run_infracost_analysis, display_cost_breakdown and fetch_resource_costs be deleted as duplicates (~143 lines).
• Expired infracost tokens now say what to do. check_infracost_authenticated() can only prove a token file exists, so an expired token passed pre-flight and failed later with raw JSON stderr. _report_scan_failure() recognises the auth case and answers infracost auth login. Why: hit this exact false positive twice while working on this.
• Smaller correctness fixes: the bigquery module description won for both the dataset and the tables, so storage rows were labelled as query cost; google_cloud_run_v2_job (teardown/offline_scoring/explainability) was in neither label map and printed as a raw google_* type; a failed infracost inspect returned [] silently, rendering as "nothing costs money" under a non-zero total; costs defaulted to workspace "development" while deploy/destroy/get-urls use "default", so it could look in a directory deploy never wrote; costs and estimate bypassed _load_config_or_exit.
• Config and docs: added roles/monitoring.viewer to REQUIRED_GCP_IAM_ROLES so doctor --project-id checks it for free (monitoring.googleapis.com was already in REQUIRED_GCP_APIS); added the cost_analysis block to config.example.yaml, which three commands read and no example documented; rewrote docs/features/costs.md, which still claimed usage-based services show $0; added estimate and costs to docs/api/cli-commands.md, where neither had appeared at all.

Sample output:

  ACTUAL COST  ·  gcp-mlops-stack-mlflow
  Measured from your real GCP usage over the last 7 days.

  Run-rate: ~$36 / month   ($34.55 always-on  +  ~$1.42 measured usage)  USD
  Charged so far: ~$8.39   (7.0 days since deploy, approximate)

  ALWAYS-ON  (billed 24/7 while the stack exists)
    $   34.55  Cloud SQL    MLflow's backend database  ·  up 100% of window

  USAGE-BASED  (measured over 7 days, scaled to a month)
    $    0.52  Cloud Run    MLflow tracking server  ·  21.4k req/mo, 240 ms median
    $    0.41  BigQuery     prediction & feature queries  ·  0.041 TB/mo scanned
    $    0.27  GCS Bucket   object / artifact storage  ·  4.82 GB stored

  You are a LIGHT user — your real usage is 0.6x the 'light' profile.
  Biggest lever: Cloud SQL is 96% of your cost and runs 24/7.

test plan

• PYTHONPATH=src pytest tests/ — 108 passing, up from 85.
• Address contract verified against a real infracost scan. This was the load-bearing assumption: measured usage is keyed on Terraform addresses, but infracost parses HCL rather than state, so if the two disagree the usage file is silently ignored and every usage line prices at $0 — the same bug, relocated. Diffed infracost's addresses (it records them in each resource's name field) against build_resource_map() on the deployed state: all 10 costable addresses match exactly, count indices included (module.experiment_tracking_mlflow[0].google_cloud_run_service.mlflow[0]). Those strings are now pinned in a test, since the tempting cleanup is to normalise the index away.
• All 9 metric/aligner pairs validated against the live Monitoring API. This caught a real bug: cloudsql.googleapis.com/database/up is GAUGE/INT64, not BOOL, so ALIGN_FRACTION_TRUE returns HTTP 400 and the uptime line degraded to a "could not read" note. ALIGN_MEAN over a 0/1 gauge is the same fraction; a test pins the aligner.
• build_resource_map() against a genuine 145KB terraform.tfstate from a real deploy. Caught that terraform show -json includes data sources (module.*.data.google_project.current), which inflated the resource map from 10 to 15; now filtered on mode == "managed".
• Resource isolation on real data: the only live GCS metric series in the test project is the _cloudbuild staging bucket, and it is correctly excluded because it isn't in the stack's state. A student's unrelated resources can't inflate their deployml cost.
• Degradation paths, each printing a fix rather than a traceback: no deployment; --days out of range; missing config; malformed config; expired infracost token (verified against a real oauth2: "invalid_grant" failure while the pre-check still returned True).
• End-to-end costs against a live deployment with real traffic. Not run — the test stack is destroyed and standing it up bills Cloud SQL. The measurement layer is verified against the live Monitoring API and the pricing layer against a real cached scan, but the two have not been exercised together on a running stack.
• deploy's new estimate prompt. Not run — needs a real deploy.

follow-up (not in this PR)

• GCS class A/B operations are inferred from method names. GCS reports methods, not billing classes, so the mapping is one we maintain. Unrecognised methods count as class B (the cheaper one) so an unknown method can't silently inflate the estimate.
• BigQuery bytes are project-scoped, not per-dataset. Scanned and stored bytes are split evenly across the deployment's BigQuery addresses. Both are priced linearly, so the total is exact and only per-row attribution is approximate — but a project with non-deployml BigQuery usage would over-attribute to the stack. INFORMATION_SCHEMA.JOBS_BY_PROJECT would fix this properly.
• Cloud Run concurrency isn't measured. Not exposed as a metric; left at infracost's default and disclosed in the output.
• No billing-export integration. Real charged dollars need a BigQuery billing export, which requires billing-account admin and has a ~24h lag — out of reach for most students, which is why this measures usage instead. The output says explicitly that it is not a billing statement.
• check_infracost_authenticated() still can't prove a token is valid without making a live call. The failure is now handled where it surfaces, but the pre-flight check remains a file-existence test.
• --compare (sqlite vs postgres) from the original proposal is still unbuilt. Given that Cloud SQL is ~96% of the bill, a one-command "what if I switched" is probably the highest-value thing left.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 58b32f77-8e37-4b41-9264-8af5963db2f3

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}