Organization-wide audit logging: a security-first CloudTrail org trail across 53 AWS accounts

Summary

We built audit and governance foundations for a 53-account AWS Organization without adopting Control Tower. Every API call in every account and Region now lands in one encrypted, immutable bucket that no workload account can touch, and the calls that matter reach Mattermost within minutes.

The story is about where things live, not which services we used. Logs are written in one account, stored in another and read from a third, so compromising any single account is not enough to change history or silence the alarm.

• Coverage: one multi-Region organization trail across all 53 accounts, encrypted with a customer-managed KMS key.
• Integrity: S3 Object Lock, run in governance mode for 30 days, then switched to compliance mode with 365-day retention.
• Access: one Terraform state per account, applied with short-lived IAM Identity Center sessions. No IAM users, no long-lived keys.
• Detection: a Lambda in the security account turns a curated set of events into Mattermost alerts, with a median of about 3 minutes from API call to alert and a p95 of about 6.
• Findings so far: three real findings in the first ten weeks. None were malicious, and each one changed a process.

The challenge

The Organization had grown to 51 accounts with no shared answer to a simple question: who did what, where, and when?

• No central audit trail. Each account had whatever CloudTrail configuration it was born with, if any. An investigation meant logging into accounts one at a time and hoping the event was still in the 90-day Event history.
• No place for security tooling to live. There was no security account and no log-archive account. Anything we built would have landed either in the management account, which should stay as empty as possible, or in a workload account, which is exactly what it is meant to watch.
• Logs were as mutable as the systems they described. Anyone with admin in an account could stop logging or delete the evidence. An attacker who gains admin usually does the same thing first.
• Nobody was told. Even where events were recorded, nothing read them until something had already gone wrong.

This project added two accounts, security and log archive, bringing the Organization to 53: the management account, the two new ones, and 50 workload accounts running production, staging and development.

Constraints, and why not Control Tower

Control Tower is the default answer to this problem, and we chose not to use it. It is a landing zone: it creates its own log-archive and audit accounts, its own trail, and a set of guardrails, and then expects to own them. Enrolling 50 existing workload accounts means accepting those baselines account by account, and accepting a service that reconciles resources we would otherwise manage in Terraform.

We needed a narrower thing: an audit foundation we fully understood, built in the same Terraform we use everywhere else, that we could explain line by line in a review. Building it by hand cost more effort up front and gave us that.

The constraints we set ourselves:

1. Everything in Terraform. No console-only configuration, so the whole design is reviewable and reproducible.
2. No long-lived credentials. Humans and pipelines authenticate through IAM Identity Center. No IAM users, no access keys on disk.
3. The management account stays thin. It holds the Organization, Identity Center and the trail definition, and nothing that processes or stores logs.
4. No single account can rewrite history. Writing, storing and reading the logs happen in different accounts with different people.
5. Additive and reversible until proven. Nothing irreversible, such as compliance-mode retention, goes on until we have watched it work.

One more constraint came from where we operate. Our workloads run in Frankfurt (eu-central-1), and the trail bucket and its key live there too. The trail records every Region, but the audit data itself never leaves the EU.

Architecture

The design follows one rule: logs are written in one account, stored in another and read from a third. CloudTrail writes from the management account, the log-archive account only stores, and the security account only reads and alerts.

The purple path is the life of one log file: an API call anywhere in the Organization, delivered by the organization trail, encrypted with the log-archive KMS key, locked in the bucket, then read by the security account and turned into an alert.

Two new accounts, two different jobs

We created a log-archive account and a security account from scratch rather than repurposing an existing one. An existing account comes with history: users, roles and trust relationships that nobody fully remembers. A new account starts with nothing, so every permission in it is one we wrote down.

The split exists to make the blast radius of any single compromise small and obvious:

• Compromised workload account: it can't stop the organization trail, because member accounts can see it but not change it. It can't reach the bucket, which lives elsewhere.
• Compromised security account: it can read logs and silence alerts, but it can't alter or delete what has already been written.
• Compromised log-archive account: Object Lock in compliance mode means even its root user can't delete a locked log version before retention ends.

Two named admins can reach the management account, both through Identity Center. Its root user sits behind a written break-glass procedure: hardware MFA, credentials in a separate vault, and a runbook that says who may open it and what to record afterwards. The member accounts have no root credentials at all; the Results section explains why.

One trail for every account and every Region

A single organization trail replaced per-account configuration. Organization trails log events for the management account and every member account, and new accounts are covered the moment they join. Member accounts can see the trail but can't modify or delete it.

It is multi-Region on purpose. A single-Region trail only records the Regions we use, and attackers like the Regions we don't: a crypto-mining fleet in a Region nobody looks at is a classic. Global service events, such as IAM and Identity Center, are captured too.

Encryption with a customer-managed KMS key. The default SSE-S3 encryption protects the disk; a customer-managed key protects the data from people. Reading a log now needs two separate grants, s3:GetObject on the bucket and kms:Decrypt on the key, and every decrypt is itself logged. The key lives in the log-archive account next to the bucket. Its policy:

• lets CloudTrail generate data keys only when the request comes from our trail, checked with the aws:SourceArn condition and the CloudTrail encryption context;
• grants kms:Decrypt to one read role in the security account, and nobody else;
• keeps key administration with the Terraform deployment role in the log-archive account.

Log file integrity validation is on. CloudTrail writes signed digest files every hour, so we can prove a log file has not been changed since delivery, independently of Object Lock.

The trail logs all management events, both read and write. The first copy of management events is free, and read calls such as ListBuckets or GetSecretValue are often the first sign of someone looking around. Data events are off for now. S3 data events on the buckets that hold customer data are the next step, once we have priced them per 100,000 events.

Terraform: one state per account, no shared secrets

Each account is its own Terraform root module with its own state. The state sits in a versioned S3 bucket inside the account it describes, with native S3 state locking (use_lockfile), so there is no shared lock table either. A plan in the security account can't see, lock or corrupt the log-archive state, and a mistake in one stack has a blast radius of one account.

Short-lived sessions only

Every apply runs on an IAM Identity Center session: aws sso login, then a named profile per account in the provider block. Sessions expire after hours, not years. There is no IAM user to forget about, no key in a CI variable, and offboarding someone is a single change in Identity Center. A leaked laptop holds, at worst, a session that is about to expire.

Values passed explicitly, not read from other states

The stacks depend on each other. The trail needs the bucket name and the key ARN from log-archive; the key policy needs the trail ARN; the alerting stack needs both. The usual shortcut is a terraform_remote_state data source that reads another account's state.

We didn't take it. Reading a state file means reading the whole file, not one output: every attribute of every resource, including anything sensitive. Granting the security account read access to the log-archive state would quietly rebuild the cross-account trust we had just separated out.

Instead, each stack exposes a few outputs, and the consuming stack takes them as typed input variables in a committed .tfvars file:

Two values break what looks like a chicken-and-egg problem. The trail ARN is deterministic from the account ID, Region and trail name, so the key policy can reference it before the trail exists. The read role is trickier, because KMS and S3 reject policies that name a principal that doesn't exist yet. The policies therefore trust the security account and pin the role with an aws:PrincipalArn condition. That keeps the apply order one-directional: log archive, then management, then security.

The cost is a manual copy whenever an output changes, which for these values is almost never. Variable validation rules catch a pasted value in the wrong format before it reaches a plan.

Object Lock: governance first, compliance once proven

The last line of defence is that logs can't be deleted at all. S3 Object Lock gives that in two modes, and the difference is whether anyone can undo it:

• Governance mode: a principal with s3:BypassGovernanceRetention can shorten retention or delete a locked version, by sending the bypass header explicitly.
• Compliance mode: nobody can shorten retention or delete a locked version before it expires. Not an admin, not the root user.

Compliance mode is the goal, and it is also a one-way door. Set the wrong retention period, or lock objects a lifecycle rule was meant to tidy up, and you pay for that storage for the full term with no way back. So we ran governance mode first and treated it as a rehearsal. During that window we checked:

1. Logs from all 53 accounts and every enabled Region were arriving under the organization prefix.
2. Digest validation passed end to end.
3. Lifecycle rules (transition to colder storage, expiry after retention) behaved correctly on locked versions.
4. The security read role could decrypt and read; nothing else could.
5. The projected storage cost at full retention was acceptable.
6. The retention period was agreed with whoever owns compliance, not picked by the engineer.

Then we switched the bucket default to compliance mode. One detail matters here: changing the default only affects objects written afterwards. Versions written during the governance phase keep governance retention until they expire, unless they are re-stamped, for example with S3 Batch Operations.

Retention is 365 days. A lifecycle rule moves log files to S3 Glacier Instant Retrieval after 90 days and expires them once retention ends. The governance phase ran for 30 days. We let its objects age out rather than re-stamping them: they expire within the year anyway, and a Batch Operations job would have cost more than the risk it removed.

From API call to Mattermost

Logs nobody reads are an archive, not a control. A Lambda function in the security account reads each new log file, matches events against a short, curated list, and posts the hits to a Mattermost channel the team already watches.

The pipeline: CloudTrail delivers a log file to the bucket, the bucket sends an object-created notification to an SQS queue in the security account, and the Lambda reads the file through the security read role, decrypting with the KMS grant described above. SQS gives us retries and a dead-letter queue without extra code. The Mattermost webhook URL is stored in Secrets Manager, not in an environment variable, and is cached for the life of each Lambda container.

Our Mattermost server only accepts traffic from allowlisted IPs, so the Lambda runs in a VPC and leaves through a NAT gateway with a fixed Elastic IP that is on the allowlist.

What we alert on

Every alert has to be something a person would act on. The list is short on purpose:

Two rules changed after the first week, which produced about 40 alerts. Single failed console sign-ins were almost all mistyped passwords, so that rule now fires only on five or more failures for one user within ten minutes. AttachRolePolicy calls made by the Terraform deployment roles are suppressed, because each of those already went through a reviewed merge request. After tuning, the channel sees four to six alerts a week.

How fast is an alert?

The floor is CloudTrail itself. AWS states that CloudTrail typically delivers logs within an average of about 5 minutes of an API call. The notification, queue and Lambda add seconds on top.

We measured 58 alerts over 30 days, from the event's eventTime to the Mattermost post timestamp. The median was 3 minutes 10 seconds, the p95 was 6 minutes 5 seconds, and the slowest was 11 minutes. The Lambda accounts for 2 to 4 seconds of that; the rest is CloudTrail delivery.

Minutes are right for an audit control. They are not right for stopping a live attack in progress. That is what the trade-off below on EventBridge is about.

Key decisions and trade-offs

Who administers the trail

Organization trails can be created in the management account or, since late 2022, by a CloudTrail delegated administrator, a member account such as security. The management account stays the owner of the trail either way.

We kept the trail in the management account. Delegating it to the security account would have reduced sign-ins to the management account, but it would also have given security the power to change or delete the trail. That breaks the rule the whole design rests on: security reads, it never writes.

Either way, the management account can still delete the trail. That risk is handled in two layers: deleting it fires an alert, and nothing already written can be removed.

Alert trigger: S3 notifications vs. EventBridge vs. CloudWatch Logs

• S3 notifications on the archive (our choice): one integration point for all accounts and Regions, and it reads exactly what the archive holds. Latency is CloudTrail's delivery time, so minutes.
• EventBridge: near real-time, but CloudTrail events reach EventBridge in the account and Region where they happen. Org-wide coverage means rules in every account and every Region, forwarding to a central bus. That is far more plumbing for faster alerts.
• CloudWatch Logs: metric filters are convenient, but the organization trail's log group sits on the management side, which would pull processing into the account we wanted to keep thin.

We picked coverage and simplicity over seconds. A fast path for a handful of critical events is on the list below.

Detective, not yet preventive

We started with detection only. An SCP that denies StopLogging, DeleteTrail and LeaveOrganization outright would prevent, but a wrong SCP can break production across every account at once, and we wanted to see real traffic first.

Six weeks after go-live, the alert history showed no legitimate use of any of those calls. We then attached an SCP to all member accounts that denies StopLogging, DeleteTrail, UpdateTrail, PutEventSelectors and LeaveOrganization. SCPs don't apply to the management account, so there the alert remains the control.

The governance-mode window

Running governance mode first left a window where a principal with the bypass permission could delete logs. We accepted it, limited the bypass permission to one Terraform role, and alerted on any bypass. The alternative was a mistake we couldn't undo for the full retention period.

A curated list, not a detection product

The Lambda enforces our own policy: for example, any CreateAccessKey is a finding because we have no IAM users. It is not anomaly detection. A managed service such as GuardDuty covers different ground and complements it rather than replacing it.

What broke

Three things went wrong during the rollout, and each one changed the design.

• The trail delivered nothing for the first hour, and the console looked fine. The KMS key policy pinned aws:SourceArn to a trail ARN built with the log-archive account ID. An organization trail's ARN carries the management account ID, so CloudTrail couldn't generate data keys. Only aws cloudtrail get-trail-status showed it, in LatestDeliveryError. We fixed the ARN and added that check to the rollout runbook.
• Member-account logs were rejected by the bucket policy. The policy allowed writes under AWSLogs/<management-account-id>/, which only covers the management account's own events. Organization trails write member-account logs under AWSLogs/<org-id>/. Adding the organization path fixed it, and the governance-mode rehearsal is where we caught it.
• One Mattermost outage meant every alert twice. One log file holds many events, so when a post failed, SQS redelivered the whole file and the alerts that had already gone out were posted again. The Lambda now records each posted eventID in a small DynamoDB table with a 24-hour TTL and skips repeats.

Results

The Organization now has one place to answer "who did what, where, and when", and it holds for every account, including ones that don't exist yet.

The seven IAM users came out of the inventory we did during the rollout. Most were old CI and backup users; each was replaced with an Identity Center profile or an IAM role.

The Saturday root sign-in. Three weeks after go-live, the root-activity rule fired for a staging account on a Saturday morning. It was a colleague updating a billing contact, using root credentials saved years earlier in a shared password manager. Nothing malicious, but it proved that member-account root credentials were still usable by anyone with vault access. We turned on centralized root access management in AWS Organizations, deleted the root credentials of all 52 member accounts, and moved the management account's root behind the break-glass procedure.

The other two findings were smaller. A developer had created an access key for a local script; it was replaced with an Identity Center profile the same day. And an admin permission set had been assigned to the wrong group in Identity Center, which we caught within minutes and corrected.

What I'd do differently

• Alert on silence, not just on events. If the trail stops delivering, the alerting Lambda simply has nothing to say. A heartbeat that fires when no log file has arrived for 15 minutes would have caught our first-hour KMS failure without anyone having to look.
• Add a fast path for the few events that can't wait. EventBridge rules for trail tampering and root sign-in only, alongside the S3 path, would bring those from minutes to seconds without rebuilding the pipeline.
• Move from detective to preventive sooner. The SCP went on after six weeks. Two would have been enough: the calls it blocks never had a legitimate use.
• Make the archive queryable from day one. Athena over the bucket, or CloudTrail Lake, turns "we have the logs" into "we can answer the question in a minute" during an investigation.
• Send cost alerts to the same channel. Budget alerts as one more subscriber would put spend in the same operational picture as security.

Sources

• Creating a trail for an organization
• CloudTrail organization delegated administrator
• Working with CloudTrail log files (delivery time)