feat: measure real GCP usage in deployml costs

config.example.yaml

@@ -27,3 +27,8 @@ stack:
       name: grafana
       params:
         service_name: grafana-server
+
+# Optional. Controls the cost commands and the estimate shown before a deploy.
+# cost_analysis:
+#   enabled: true            # show a cost estimate in the deploy confirmation
+#   warning_threshold: 100.0 # warn in USD/month above this figure

docs/api/cli-commands.md

@@ -59,6 +59,49 @@ Builds run on Cloud Build, so a local Docker daemon is not required for GCP mode
 
 ---
 
+## `deployml estimate`
+
+Predict monthly cost from your config, without deploying or touching any cloud
+resources. Reads the Terraform deployml would generate and prices it.
+
+```bash
+deployml estimate
+deployml estimate --profile heavy
+```
+
+**Options:**
+- `--config-path`, `-c`: Path to config YAML. Default `config.yaml`.
+- `--profile`, `-p`: Usage assumption, `light` (default) or `heavy`. Light is a
+  typical student working through the coursework; heavy is a busy course project,
+  roughly 10-20x the traffic.
+
+Infracost prices usage-based services at zero usage by default, which would report
+Cloud Run, BigQuery, and GCS at $0 and make the estimate look far cheaper than
+reality. The profile supplies a realistic usage assumption instead, and the output
+separates the two kinds of cost:
+
+```
+  ~$35 / month   ($34.55 fixed  +  ~$0.92 usage)  USD
+
+  ALWAYS-ON  (billed 24/7 even if you never use the stack)
+    $   34.55  Cloud SQL    MLflow's backend database
+
+  USAGE-BASED  (scales with activity · profile: light)
+    $    0.44  BigQuery     prediction & feature queries
+    $    0.24  GCS Bucket   MLflow model artifacts
+    $    0.24  Cloud Run    MLflow tracking server
+
+  Biggest lever: Cloud SQL is 97% of your cost and runs 24/7.
+```
+
+Changing the profile moves only the usage line; the always-on baseline does not
+move, which is the point. Not supported for GKE deployments.
+
+Requires infracost installed and authenticated (`infracost auth login`), but no
+cloud credentials.
+
+---
+
 ## `deployml deploy`
 
 Deploy infrastructure from a YAML config file. Prompts for confirmation by default.
@@ -80,6 +123,58 @@ First-time deploy takes about 20 minutes because Cloud SQL Postgres provisioning
 
 ---
 
+## `deployml costs`
+
+Show what a running deployment actually costs, measured from real GCP usage
+rather than assumed.
+
+```bash
+deployml costs
+deployml costs --days 7
+```
+
+**Options:**
+- `--config-path`, `-c`: Path to config YAML. Default `config.yaml`.
+- `--days`, `-d`: How many days of real usage to measure, 1-90. Default 30. If the
+  deployment is younger than the window, the window shrinks to the deployment's age
+  so the run-rate is not diluted by days that never happened.
+
+Where `estimate` has to assume your usage, this reads it: requests served per Cloud
+Run service, bytes stored in GCS, bytes scanned by BigQuery, and how much of the
+window the database was up. Those measurements are pulled from Cloud Monitoring and
+priced through infracost, so each line shows the number that produced it:
+
+```
+  ACTUAL COST  ·  gcp-mlops-stack
+  Measured from your real GCP usage over the last 7 days.
+
+  Run-rate: ~$36 / month   ($34.55 always-on  +  ~$1.42 measured usage)  USD
+  Charged so far: ~$8.40   (7.0 days since deploy, approximate)
+
+  ALWAYS-ON  (billed 24/7 while the stack exists)
+    $   34.55  Cloud SQL    MLflow's backend database  ·  up 100% of window
+
+  USAGE-BASED  (measured over 7 days, scaled to a month)
+    $    0.71  Cloud Run    MLflow tracking server  ·  30.0k req/mo, 240 ms median
+    $    0.44  BigQuery     prediction & feature queries  ·  0.041 TB/mo scanned
+
+  You are a LIGHT user — your real usage is 0.6x the 'light' profile.
+```
+
+Anything Cloud Monitoring cannot report is priced at $0 and listed under
+"About these numbers", along with the places a measurement is an approximation,
+so a zero is always explained rather than silently assumed.
+
+This is not a billing statement. It prices your measured usage at list rates;
+for the amount you were actually charged, see the GCP Billing Console.
+
+**Requires:**
+- infracost installed and authenticated
+- Application Default Credentials: `gcloud auth application-default login`
+- `roles/monitoring.viewer` on the project (checked by `deployml doctor --project-id`)
+
+---
+
 ## `deployml get-urls`
 
 Print service URLs from the last deployment and write them to a `.env` file. Database credentials are masked.

docs/features/costs.md

@@ -1,46 +1,168 @@
 # Cost Estimates
 
-deployml integrates with [Infracost](https://github.com/infracost/infracost) to provide cost estimates before deploying your infrastructure, helping you manage cloud costs effectively in academic settings.
+deployml uses [Infracost](https://www.infracost.io) to answer two different
+questions about money, with one command each:
 
-## Overview
+| Command | Question | Where the usage numbers come from |
+|---|---|---|
+| `deployml estimate` | What *will* this cost? | A light/heavy assumption, since the stack does not exist yet |
+| `deployml costs` | What *is* this costing me? | Cloud Monitoring — your real requests, bytes, and uptime |
 
-Cost analysis runs automatically during deployment, showing monthly cost estimates for your entire stack, cost breakdowns by component, and warnings if costs exceed your configured threshold. The process analyzes your Terraform configuration before deployment, allowing you to adjust your configuration based on estimates.
+Neither is a billing statement. Both price resources at list rates; for the amount
+you were actually charged, check the GCP Billing Console.
 
 ## Setup
 
-Install Infracost and register for a free API key using the instructions [here](https://www.infracost.io/docs/#quick-start). 
+```bash
+brew install infracost
+infracost auth login
+```
 
-## Configuration
+`deployml costs` additionally needs Application Default Credentials and read access
+to metrics:
+
+```bash
+gcloud auth application-default login
+```
 
-Cost analysis is enabled by default. Configure it in your YAML file to enable or disable cost analysis, set a warning threshold in USD (default $100/month), and choose the currency for cost display.
+Run `deployml doctor --project-id YOUR_PROJECT` to confirm infracost is installed and
+authenticated and that you hold `roles/monitoring.viewer`.
 
-Here is an example of what this might look like:
-```yaml
-cost_analysis:
-  enabled: true              # Enable/disable cost analysis (default: true)
-  warning_threshold: 50.0    # Warn if monthly cost exceeds this amount (default: 100.0)
-  currency: "USD"   
-  bucket_amount: 200        # GB stored across GCS buckets
-  cloudsql_amount: 50       # GB of Cloud SQL storage
+## Fixed cost vs usage cost
+
+This is the distinction that makes the numbers make sense, and both commands are
+built around it.
+
+- **Always-on** resources bill 24/7 for existing, whether or not anyone touches
+  them. Cloud SQL is the one that matters here.
+- **Usage-based** resources bill per request, per GB, per byte scanned. Cloud Run,
+  BigQuery, and GCS all scale to zero.
+
+Infracost assumes **zero usage** unless you tell it otherwise. That is why a naive
+estimate reports $0 for every usage-based service and makes the stack look free
+apart from the database. Both commands fix this, in different ways.
+
+## Before deploying: `deployml estimate`
+
+```bash
+deployml estimate
+deployml estimate --profile heavy
 ```
 
-## Typical Costs
+Renders the Terraform your config would produce, prices it against a usage profile,
+and splits the result:
 
-Here are estimated typical costs for several **GCP** services, but please do not simply believe these numbers without keeping track of costs yourself.
+```
+  ~$35 / month   ($34.55 fixed  +  ~$0.92 usage)  USD
 
-- Cloud Run services cost $10-30 per month depending on traffic.  
-- Cloud SQL PostgreSQL ranges from $7/month for small instances to $25+ for production.  
-- Google Cloud Storage costs approximately $0.020 per GB per month.   
-- BigQuery storage costs $0.020 per GB per month with query costs based on data scanned.  
-- Cloud VMs cost approximately $25 per month for medium instances.   
-- GKE clusters have no management fee, but you pay for VM instances and load balancers. MLflow on GKE also provisions a small PersistentDisk for its data, a few cents per GB-month. Note that GKE can get expensive quickly.
+  ALWAYS-ON  (billed 24/7 even if you never use the stack)
+    $   34.55  Cloud SQL    MLflow's backend database
 
+  USAGE-BASED  (scales with activity · profile: light)
+    $    0.44  BigQuery     prediction & feature queries
+    $    0.24  GCS Bucket   MLflow model artifacts
+    $    0.24  Cloud Run    MLflow tracking server
 
+  Biggest lever: Cloud SQL is 97% of your cost and runs 24/7.
+  Switch MLflow to a SQLite backend to drop this to ~$0/month.
+```
 
-## Cost Optimization
+`light` is a student working through the coursework; `heavy` is a busy course
+project at roughly 10-20x the traffic. Switching profiles moves only the usage line
+— the always-on baseline does not budge. That is the lesson: how hard you use the
+stack barely matters next to whether the database is running.
+
+No cloud credentials needed, and nothing is deployed.
+
+## While deployed: `deployml costs`
+
+```bash
+deployml costs
+deployml costs --days 7
+```
+
+Once the stack exists we can stop guessing. This reads your real usage out of Cloud
+Monitoring — requests per service, bytes stored, bytes scanned, database uptime —
+and prices that instead:
+
+```
+  ACTUAL COST  ·  gcp-mlops-stack
+  Measured from your real GCP usage over the last 7 days.
+
+  Run-rate: ~$36 / month   ($34.55 always-on  +  ~$1.42 measured usage)  USD
+  Charged so far: ~$8.40   (7.0 days since deploy, approximate)
+
+  ALWAYS-ON  (billed 24/7 while the stack exists)
+    $   34.55  Cloud SQL    MLflow's backend database  ·  up 100% of window
 
-Here are some tips to keep the costs low while you are learning:  
+  USAGE-BASED  (measured over 7 days, scaled to a month)
+    $    0.71  Cloud Run    MLflow tracking server  ·  30.0k req/mo, 240 ms median
+    $    0.44  BigQuery     prediction & feature queries  ·  0.041 TB/mo scanned
+
+  You are a LIGHT user — your real usage is 0.6x the 'light' profile.
+
+  Biggest lever: Cloud SQL is 96% of your cost and runs 24/7.
+  Tear the stack down when you are not using it:  deployml destroy
+```
+
+Every usage line carries the measurement behind it, so you can see *why* a number
+is what it is. Once you have run this, `light` and `heavy` stop being assumptions
+you are billed for and become a label on your actual behaviour.
+
+Counters (requests, bytes scanned) are summed over the window and scaled to 30
+days. Levels (bytes stored) are averaged over the window. Anything Monitoring does
+not report is priced at $0 and listed under "About these numbers", together with
+the places where a measurement is an approximation — a zero is always explained,
+never silently assumed.
+
+## Cost shown during deploy
+
+`deployml deploy` runs the usage-aware estimate after `terraform plan` and shows the
+total before the confirmation prompt:
+
+```
+ Deploy stack? Monthly cost: ~$35.47 USD [y/N]:
+```
+
+If infracost is missing or unauthenticated the estimate is skipped and the deploy
+continues.
+
+## Configuration
+
+```yaml
+cost_analysis:
+  enabled: true             # set to false to skip the estimate during deploy (default: true)
+  warning_threshold: 50.0   # warn if monthly cost exceeds this (default: 100.0)
+```
+
+## Typical costs (Cloud Run stack)
+
+A standard MLflow + FastAPI + Grafana deployment runs around **$35/month**, and
+roughly 97% of that is the Cloud SQL instance. The usage-based services add well
+under a dollar at student traffic levels.
+
+- Cloud SQL PostgreSQL: from ~$7/month for the smallest instance to $25+ for
+  production sizes. Billed continuously.
+- Cloud Run: scales to zero, so a few cents per month at light traffic; $10-30/month
+  under sustained load.
+- Google Cloud Storage: ~$0.020 per GB per month.
+- BigQuery: ~$0.020 per GB per month storage, plus query cost per byte scanned.
+- Cloud VMs: ~$25/month for medium instances.
+- GKE: no management fee for one zonal cluster, but you pay for the VM instances and
+  load balancers, and MLflow on GKE provisions a small PersistentDisk. This adds up
+  quickly.
+
+## Cost Optimization
 
-- Use SQLite instead of Cloud SQL whenever possible, particularly for development purposes and when your data is small. The minikube and GKE MLflow paths already do this, sqlite on a PersistentVolumeClaim, so they avoid the always-on Cloud SQL cost.  
-- Enable auto-teardown to prevent forgotten deployments. 
-- Use Cloud Run for variable workloads to take advantage of scale-to-zero pricing.
+Tips to keep costs low while you are learning:
+
+- **Use SQLite instead of Cloud SQL** where you can. Set
+  `backend_store_uri: sqlite` instead of `postgresql` to remove the Cloud SQL
+  instance entirely — that is the single biggest saving available, since it deletes
+  ~97% of the bill. The minikube and GKE MLflow paths already do this, with sqlite on
+  a PersistentVolumeClaim.
+- **Always run `deployml destroy`** when you are done. Cloud SQL bills continuously,
+  used or not. `deployml costs` reports database uptime so you can see how much of
+  the month you actually left it running.
+- **Enable auto-teardown** so a forgotten deployment cannot bill all semester.
+- **Prefer Cloud Run for variable workloads** to take advantage of scale-to-zero.

docs/tutorials/gcp-cloud-run.md

@@ -6,6 +6,13 @@ This tutorial walks you through deploying a full MLOps stack on GCP using Cloud
 
 Run `deployml doctor --project-id YOUR_GCP_PROJECT_ID` first. It checks everything in the [installation guide](../installation.md). Make sure the IAM and API checks pass before you continue.
 
+Infracost is optional. Install it only if you want the cost estimate steps below:
+
+```bash
+brew install infracost
+infracost auth login
+```
+
 ## 1. Create and prepare your GCP project
 
 1. Create a project in the [GCP Console](https://console.cloud.google.com). Note the project ID.
@@ -63,6 +70,16 @@ stack:
 - `model_serving` deploys a FastAPI container that pulls the latest registered model from MLflow on startup.
 - `model_monitoring` deploys Grafana connected to the Postgres `metrics` database.
 
+## 3.5 Estimate costs (optional)
+
+Check what the stack will cost before committing to a 20-minute deploy:
+
+```bash
+deployml estimate
+```
+
+No GCP credentials required, no infrastructure touched. A standard Cloud Run stack runs around **$34/month** — almost entirely Cloud SQL. See [Cost Estimates](../features/costs.md).
+
 ## 4. Build Docker images
 
 ```bash
@@ -159,6 +176,14 @@ You should see `offline_features`, `predictions`, `ground_truth`, and `drift_met
 
 With the stack running, follow the [example walkthrough](example.md) to train a model, register it, serve predictions through FastAPI, and visualize drift metrics in Grafana.
 
+## 9.5 Check running costs
+
+```bash
+deployml costs
+```
+
+Shows what your deployed stack is currently costing. Cloud SQL is the main driver at ~$34/month — everything else scales to zero.
+
 ## 10. Teardown
 
 When you are done, destroy all infrastructure to stop billing: