Document HfLA AWS billing alerting

[ale210]

Dependency

• Consolidate AWS billing alerting into the organization account #175 — the page should document the end state, not the configuration being replaced.

Overview

We need to document how HfLA's AWS billing alerting works, because none of it exists in Terraform and so the configuration is recorded nowhere. Today the only way to find out what alerts exist, what they are set to, or who receives them is to log into the AWS console and look.

Action Items

• Write a wiki page covering the following. It should be readable by someone who has never seen the AWS account.
• Where billing alerting lives, and why. All billing alerts and budgets belong in the organization (payer) account 504695638931 only, never in a member account. Give the reason, not just the rule: under consolidated billing the AWS/Billing EstimatedCharges metric is published only in the payer account and already covers every member account, so a CloudWatch billing alarm created in a member account has no data and sits at INSUFFICIENT_DATA indefinitely. AWS Budgets are the exception in that they do function per-account, which is why one existed in the incubator account and had to be removed — so the rule needs stating explicitly rather than being left to infer.
• The account layout. Org o-x3897m6l40, payer 504695638931 (hackforla), members 035866691871 (incubator) and 470363915259 (foodoasis). Note that billing metrics are only in us-east-1.
• What is currently configured — the budgets, their thresholds, and the reasoning behind those numbers. Link back to the decision so the next person does not have to guess whether a threshold is deliberate or stale. The alarm being replaced had sat at $210 while actual spend ran higher every month, and nobody could tell whether that was intentional.
• Who receives alerts and how to change that. Currently the SNS topic Monthly_Org_Billing_Alarm with a single subscriber, hackforla@hackforla.org. Include how to add or remove a subscriber and the fact that new email subscriptions must be confirmed before they deliver.
• How to bring a new member account under billing alerting — create a LinkedAccount-filtered budget in the org account; do not create anything in the new account itself.
• What to do when an alert fires — how to see what drove the spend, and who to tell.
• After the page is published, ask someone who has not worked on this to read it and confirm they could act on an alert using only the page. Fix whatever they get stuck on.

Note: this depends on the implementation issue being finished first, since the page should document the end state rather than the current one. If it turns out useful to draft in parallel, write it against the intended end state and confirm the details before publishing.

Resources/Instructions

• Research and findings: Research - AWS billing alerts incubator#118 (see the findings comment)
• Implementation issue: Consolidate AWS billing alerting into the organization account #175
• foodoasis account audit: Audit the foodoasis AWS account for billing alarms and budgets #176 — its result determines whether that account needs mentioning as covered or as an exception.
• Target: the DevOps wiki. Note the wiki is a separate git repo from the code repo and gets edited by other people through the web UI, so pull immediately before editing and again before pushing.
• Consolidated billing and where billing data appears: https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html
• https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html