Treating cloud spend like uptime: cost alerting across an AWS Organization

Summary

We made AWS spend across the Organization observable and alertable, the same way we already treat uptime: thresholds, forecasts, routing and an owner for every alert. Forecasted alerts now warn us about 9 days before a budget is breached.

The work has two layers. An alerting layer, built as a reusable Terraform module on AWS Budgets, tells us when spend is heading somewhere it shouldn't. A visibility layer, an audit method that joins billing data with resource inventory, tells us why.

The same lens then shaped a decision inside observability itself: whether ECS logs go to CloudWatch Logs or to Loki via FireLens.

The challenge

The AWS Organization has a management account and member accounts. Uptime had alerts, dashboards and on-call routing. Spend had none of that: the first signal of a cost problem was the monthly invoice, weeks after the money was gone.

That gap has a predictable shape. A forgotten NAT Gateway, a log group with no retention, or a test environment nobody tore down costs a little each day. None of it is dramatic enough to notice in the console, and all of it compounds. In one month, a load-test environment left running in a staging account added roughly $1,100 to a bill of about $9,500, and nobody noticed until the invoice arrived.

The goal was to treat cost as an operational signal: measured continuously, alerted on before it breaches, and owned by someone.

Constraints

1. Warn before, not after. An alert that fires once the budget is already spent is a post-mortem, not an alert. Forecasting was a hard requirement.
2. One module, multiple accounts. Budgets had to be defined once in Terraform and stamped out per account, service or tag, not clicked together in the console.
3. Routing without rework. Adding a new subscriber, such as a chat channel or an on-call tool, must not mean touching or recreating the budget itself.
4. No noisy plans. The module had to produce clean terraform plan output month after month, so real drift stays visible.
5. AWS-native first. Use Budgets, Cost Explorer, the tagging APIs and SNS before reaching for third-party FinOps tooling.

Architecture

The system has two layers joined by one feedback loop. The alerting layer says when spend is heading off plan; the visibility layer says why, and its tags flow back into the alerting layer as budget filters.

Everything on the alerting side is Terraform. The audit runs against the Cost Explorer and tagging APIs, and its lasting output is a tag schema rather than a report.

The alerting layer: a reusable Budgets module

The alerting layer is one Terraform module built on AWS Budgets, instantiated per account, per service or per tag.

Why Budgets, not CloudWatch billing alarms

The classic approach is a CloudWatch alarm on the EstimatedCharges metric. It has two problems. Billing metrics only exist in us-east-1, so every alarm has to live there regardless of where workloads run. And the alarm only fires once actual charges cross the line; it has no forecast.

AWS Budgets supports both actual and forecasted thresholds, filters by service, tag or linked account, and plain monitoring and notifications are free.

Thresholds: three actual, one forecasted

The forecasted notification is the one that changes behaviour. It turns cost from something you review into something you are warned about, while there is still time to act.

SNS fan-out, and the policy most modules get wrong

Budgets publishes to an SNS topic, and subscribers hang off that topic. Budgets cannot publish to a topic unless the topic policy explicitly allows the budgets.amazonaws.com service principal. Without it, the budget creates fine, the notification is configured, and nothing is ever delivered.

Putting the topic between the budget and its subscribers also decouples them: a new email list, chat webhook or on-call integration is a new subscription, and the budget resource is never touched.

Scoping

The module accepts a map of cost filters, so the same code covers a whole linked account, a single service such as Amazon EC2, or a cost allocation tag such as a team or environment.

A simplified version of the core resources:

data "aws_iam_policy_document" "billing_topic" {
  statement {
    sid       = "AllowBudgetsPublish"
    actions   = ["sns:Publish"]
    resources = [aws_sns_topic.billing.arn]

    principals {
      type        = "Service"
      identifiers = ["budgets.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
  }
}

resource "aws_sns_topic_policy" "billing" {
  arn    = aws_sns_topic.billing.arn
  policy = data.aws_iam_policy_document.billing_topic.json
}

locals {
  notifications = [
    { type = "ACTUAL", threshold = 50 },
    { type = "ACTUAL", threshold = 80 },
    { type = "ACTUAL", threshold = 100 },
    { type = "FORECASTED", threshold = 100 },
  ]
}

resource "aws_budgets_budget" "this" {
  name              = var.name
  budget_type       = "COST"
  limit_amount      = var.limit_amount
  limit_unit        = "USD"
  time_unit         = "MONTHLY"
  time_period_start = "2024-01-01_00:00" # pinned in the past, see trade-offs

  dynamic "cost_filter" {
    for_each = var.cost_filters # e.g. { LinkedAccount = ["111111111111"] }
    content {
      name   = cost_filter.key
      values = cost_filter.value
    }
  }

  dynamic "notification" {
    for_each = local.notifications
    content {
      comparison_operator       = "GREATER_THAN"
      threshold                 = notification.value.threshold
      threshold_type            = "PERCENTAGE"
      notification_type         = notification.value.type
      subscriber_sns_topic_arns = [aws_sns_topic.billing.arn]
    }
  }

  depends_on = [aws_sns_topic_policy.billing]
}

The visibility layer: billing data meets inventory

Alerts say that spend is off; they don't say why. The audit method that answers why reads two sources and compares them.

• What you pay for: Cost Explorer and its API, grouped by account, service and usage type.
• What exists: the Resource Groups Tagging API, which lists tagged resources across services and regions.

Neither view is enough on its own, because AWS bills by usage, not by inventory. Data transfer, NAT processing and log ingestion appear on the bill but are not resources you can list. An unattached volume or an idle load balancer is a resource, but its cost blends into a service line that looks normal. The useful findings sit where the two views disagree.

The usual suspects

Tags as ongoing governance

The audit is a snapshot; tags keep the picture current. A small, enforced set of cost allocation tags makes every future question cheaper to answer: which team owns this, and is it production? Activated tags also become budget filters in the module above, so a team can get its own budget without a new account.

The schema is four keys: owner, environment, service and cost-center. The Terraform provider's default_tags applies them to everything we create, an AWS Organizations tag policy reports non-compliant resources, and a CI check fails any plan that would create an untagged resource.

Observability has a cost curve too

Once spend is visible, it starts shaping decisions about observability itself. The clearest case was where the ECS services send their logs.

• CloudWatch Logs has no infrastructure to run and integrates natively with ECS. Its cost is almost entirely per GB ingested, so it grows linearly with log volume.
• FireLens to Loki routes logs through a Fluent Bit sidecar into Loki, which already runs as part of our LGTM stack and stores chunks in S3. Its cost is mostly fixed: the Loki footprint, cheap object storage, and a small CPU and memory tax per task for the sidecar.

At low volume, CloudWatch is cheaper and simpler. Past a certain daily ingestion volume, the fixed cost of Loki is smaller than the per-GB bill. We made the choice per service on that crossover point rather than as a blanket rule.

The crossover landed at roughly 3 GB of logs per service per day. At Frankfurt's ingestion price of about $0.63 per GB, that is around $57 a month per service in CloudWatch, against a few dollars of sidecar and S3 cost on a Loki stack we already run. Three of our 12 ECS services sat above that line and moved to FireLens; the other nine stayed on CloudWatch Logs.

CloudWatch's Infrequent Access log class narrows the gap for logs that are rarely queried, at the price of features such as metric filters. It is worth pricing as a third option before moving a service.

Key decisions and trade-offs

Where budgets live

Budgets can sit centrally in the management account, one per member account via a LinkedAccount filter, or inside each member account. Central budgets give one place to see coverage and one Terraform state, but member-account teams can't see them. Per-account budgets put ownership where the spend happens, at the cost of multiple deployments to keep consistent.

Price is not the deciding factor. Budgets used only for monitoring and notifications are free; only action-enabled budgets are charged after the first two.

We run a hybrid. Every member account has one budget held centrally in the management account with a LinkedAccount filter, which gives full coverage from a single Terraform state. The three production accounts also carry their own per-service budgets, owned by the teams that run those services.

Tag lag

A cost allocation tag is not useful the moment you add it. A new tag key can take up to 24 hours to show up for activation in the management account, and roughly another 24 hours after activation to appear in Cost Explorer and budget filters. Tag-scoped budgets therefore start blind. The management account can backfill tag activation up to twelve months, which recovers history but not the alerts you didn't get.

Fixed thresholds vs. Cost Anomaly Detection

Fixed thresholds answer one question: will we exceed the plan? They miss a spike that stays under budget, such as a service whose daily cost triples in a month where the account still lands at 70%. Cost Anomaly Detection models normal spend per service or account and flags deviations regardless of budget. The two are complementary: budgets for the plan, anomaly detection for the shape.

Drift: pinning time_period_start

If time_period_start is left unset, AWS fills it with the start of the current period. The next month, Terraform sees a different value and proposes a change on every plan. Pinning it to a fixed past date keeps plans clean, so a diff on a budget always means something changed.

What broke

• Silent notifications. A budget with a notification pointing at an SNS topic looks complete in both Terraform and the console. Without the topic policy for budgets.amazonaws.com, nothing is delivered and nothing errors. The fix went into the module, with an explicit depends_on so the policy exists before the budget.
• A plan that never settled. Leaving time_period_start to AWS meant every new month produced a diff on every budget. Pinning the date fixed it, but only after the noise had trained people to skim past budget diffs.
• Tag-scoped budgets that read zero. Budgets filtered on a freshly activated tag reported no spend for the first day or two. Nothing was wrong; the tag just wasn't in billing data yet.

None of the three cost money directly, but the first one meant our earliest budgets were silently not alerting for about two weeks.

Results

Cost now behaves like any other production signal: it has thresholds, a forecast, an alert route and an owner.

The clearest catch was in a staging account, a small account within the Organization. After an incident, debug-level logging was left on in one ECS service, and CloudWatch Logs ingestion jumped from under 1 GB to about 40 GB a day. On day 9 of the month, the forecasted alert for that account fired in the shared channel, and the log level was reverted the same afternoon. Left until the invoice, it would have added roughly $550 to the month.

What I'd do differently

• Route budget alerts to Mattermost from day one. CloudTrail security alerts already land in Mattermost. Putting cost alerts in the same place, as one more SNS subscriber, would make them part of the daily operational picture rather than an email folder.
• Turn on Cost Anomaly Detection alongside the budgets. Fixed thresholds were the right first step, but they can't see a spike that stays under plan.
• Set up Cost and Usage Reports with Athena early. Cost Explorer answers the common questions; CUR with Athena answers the awkward ones, such as cost per task or per tenant, with SQL and full history.
• Agree the tag schema before the audit. Tag lag means every day without activated tags is a day of spend you can't attribute later without a backfill.