Ephemeral per-branch environments with GitOps and zero static credentials

Summary

Every merge request now gets its own live environment: a namespace, a URL and a TLS certificate, created when the MR opens and destroyed when it merges or closes. Reviewers click a link instead of checking out a branch, and nothing in the delivery path holds a long-lived AWS key.

The interesting part is not that preview environments exist. It is that they are reconciled rather than scripted: the set of running environments is derived from the set of open MRs, so a missed webhook or a failed pipeline cannot leave one behind.

• Spin-up: MR opened to environment serving HTTPS in about 4.5 minutes at p50 and 8 minutes at p95.
• Teardown: MR merged or closed to namespace, certificate and DNS records gone in about 1.5 minutes at p50 and 3 minutes at p95.
• Volume: about 340 environments created since go-live in February 2026, roughly 9 a week.
• Cleanup reliability: zero orphaned namespaces and zero orphaned Route53 records at the October 2026 audit.
• Credentials: zero static AWS keys. cert-manager exchanges a Kubernetes service account token for short-lived STS credentials on every DNS change.

The challenge

Review was slow because there was only one place to see a change running, and everyone wanted it at once.

• One shared staging environment. Branches queued for it. Deploying your MR meant overwriting someone else's, so reviewers often approved from the diff alone.
• Behaviour was reviewed last. Visual and integration problems surfaced after merge, on dev, where they were more expensive to unpick and blocked everyone else's work.
• Manual environments did not clean up after themselves. Anything spun up by hand outlived the branch it was built for.
• DNS automation meant a stored key. The usual way to let a cluster manage Route53 records is an IAM user access key in a Kubernetes Secret: long-lived, rarely rotated, and readable by anyone who can read that Secret.

Before previews, the median MR took about 26 hours from opening to approval, and roughly a third of that was spent waiting for a turn on staging.

Architecture

Two independent loops meet at the image tag. GitLab CI builds and pushes an image tagged with the MR's head commit. Argo CD, separately, asks GitLab which MRs are open and renders one Application per MR that expects exactly that tag.

Neither side calls the other. CI never touches the cluster, and Argo CD never runs a build. That separation is what lets the cluster hold no CI credentials and lets CI hold no cluster credentials.

1. A developer opens an MR. The pipeline builds the image with rootless BuildKit and pushes it to the GitLab Container Registry as <short-sha>, authenticating with the job's own CI_JOB_TOKEN.
2. The ApplicationSet controller sees the new MR on its next poll or webhook and generates an Application named after the MR number.
3. Argo CD syncs the Application into its own namespace: Deployment, Service, routing and a Certificate.
4. cert-manager solves a DNS-01 challenge in Route53 using short-lived credentials, and the environment comes up on mr-<iid>.preview.<domain> over HTTPS.
5. When the MR merges or closes, it drops out of the generator's result, the Application is deleted, and its finalizer removes everything it created, including the namespace.

The cluster is self-managed (kubeadm), not EKS, which matters for how AWS credentials work in Decision 3.

Decision 1: Argo CD ApplicationSets with the pull request generator

We made environments a function of open MRs, not a side effect of pipeline jobs. The pull request generator queries the GitLab API for open MRs and produces one parameter set per MR; the ApplicationSet turns each set into an Application.

The obvious alternative was GitLab's own review apps: a deploy job on MR pipelines and an on_stop job to tear down. We rejected it for two reasons.

• Teardown depends on a job running. If the stop job fails, is cancelled, or the runner is unavailable when the branch is deleted, the environment is orphaned and nothing notices.
• CI would need cluster credentials. A push-based deploy means every pipeline holds a kubeconfig, which is exactly the kind of static credential we were trying to remove.

With the generator, deletion is reconciliation. A closed MR simply stops appearing in the result, and the controller removes its Application on the next pass.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: previews
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - pullRequest:
        requeueAfterSeconds: 60
        gitlab:
          project: "<project-id>"
          pullRequestState: opened
          tokenRef:
            secretName: gitlab-mr-reader
            key: token
  template:
    metadata:
      name: "preview-mr-{{ .number }}"
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: previews
      source:
        repoURL: <gitops-repo>
        targetRevision: main
        path: charts/app
        helm:
          parameters:
            - name: image.tag
              value: "{{ .head_short_sha }}"
            - name: host
              value: "mr-{{ .number }}.preview.<domain>"
      destination:
        server: https://kubernetes.default.svc
        namespace: "preview-mr-{{ .number }}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Three details carry most of the reliability:

• The namespace is rendered by the chart, not by CreateNamespace=true. A namespace created through that sync option is not tracked by the Application, so cascade deletion leaves it behind. Rendering the Namespace object makes Argo CD own it, and it is deleted with everything else.
• The finalizer makes deletion cascade. Without resources-finalizer.argocd.argoproj.io, deleting the Application would orphan its workloads.
• The tag contract is exact. Argo CD's head_short_sha is 8 characters, matching GitLab's CI_COMMIT_SHORT_SHA, so CI and Argo CD agree on the tag without talking to each other. A new push to the MR changes the tag, and the environment follows it.

Previews live in a dedicated Argo CD project that only allows preview-mr-* destinations, so a bad template cannot reach dev or prod namespaces. A GitLab merge request webhook points at the ApplicationSet controller, so new and closed MRs are picked up within seconds; the 60-second poll is only a fallback for missed events.

Decision 2: GitLab CI with rootless BuildKit

We build images with rootless BuildKit because the default answer, Kaniko, no longer has an upstream. Google archived the GoogleContainerTools/kaniko repository in June 2025, and its gcr.io/kaniko-project images stopped receiving updates. A Chainguard-maintained fork exists, but starting a new pipeline on a fork of an archived project means inheriting someone else's maintenance decisions.

BuildKit is the engine behind docker build, so Dockerfile behaviour matches what developers see locally. It also gives us registry-backed layer caching, which matters when every push to an MR triggers a build.

build:
  image:
    name: moby/buildkit:rootless
    entrypoint: [""]
  variables:
    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
  before_script:
    - mkdir -p ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf '%s:%s' "$CI_REGISTRY_USER" "$CI_JOB_TOKEN" | base64 | tr -d '\n')\"}}}" > ~/.docker/config.json
  script:
    - buildctl-daemonless.sh build
      --frontend dockerfile.v0
      --local context=. --local dockerfile=.
      --output type=image,name=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA,push=true
      --export-cache type=registry,ref=$CI_REGISTRY_IMAGE:buildcache,mode=max
      --import-cache type=registry,ref=$CI_REGISTRY_IMAGE:buildcache

Two trade-offs to be honest about:

• --oci-worker-no-process-sandbox lets BuildKit run without creating its own PID namespace, which the runner pod's default security profile does not allow. RUN steps then share the build container's process namespace. For a single-purpose build pod that is acceptable; for multi-tenant runners it deserves a closer look.
• The registry credential is the job token. CI_JOB_TOKEN exists only for the life of the job, so the push path adds no stored secret.

Decision 3: cert-manager DNS-01 against Route53, with no stored AWS keys

Each environment gets a real certificate through an ACME DNS-01 challenge, and cert-manager authenticates to AWS with a Kubernetes service account token instead of an access key.

Why DNS-01. HTTP-01 needs the new hostname to be routable before the certificate exists, and it ties issuance to whichever ingress controller is serving the challenge. DNS-01 only needs a TXT record, so issuance does not depend on routing, works for wildcard names, and survived the ingress controller change described below untouched.

Why no keys, and how on a non-EKS cluster. EKS hands you IRSA. A kubeadm cluster does not, but the mechanism underneath is plain OIDC, so we rebuilt it:

1. The API server's service account issuer is set to a public HTTPS URL, and its discovery document and signing keys (/.well-known/openid-configuration and the JWKS) are published there.
2. That URL is registered in IAM as an OIDC identity provider.
3. An IAM role trusts that provider, with conditions pinning the audience to sts.amazonaws.com and the subject to cert-manager's service account.
4. The ClusterIssuer references that service account. cert-manager requests a short-lived, audience-bound token for it and calls sts:AssumeRoleWithWebIdentity itself (cert-manager Route53 docs).

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-dns01-account
    solvers:
      - dns01:
          route53:
            region: <region>
            hostedZoneID: <preview-zone-id>
            role: arn:aws:iam::<account-id>:role/cert-manager-dns01
            auth:
              kubernetes:
                serviceAccountRef:
                  name: cert-manager-route53

The role's permissions are scoped as tightly as Route53 allows. ChangeResourceRecordSets is limited to the preview hosted zone, and IAM condition keys restrict it to TXT records whose names start with _acme-challenge.. A stolen token, if one ever leaked, could only write challenge records in one zone, and only until its STS session expires.

One limit to plan for. Let's Encrypt allows 50 new certificates per registered domain every 7 days. A certificate is issued once per MR, not per push, so our busiest week, 19 new MRs, used under 40% of the budget. A team pushing past roughly 40 MRs a week should switch to a single wildcard certificate for *.preview.<domain>, which DNS-01 already supports.

Decision 4: dev tracks main, prod is pinned to tags

Previews answer "does this change work?", dev answers "does main work?", and prod only ever runs a version someone deliberately chose. Each tier differs in what it follows and who can move it.

A prod release is a one-line diff in the GitOps repo: targetRevision moves from one tag to the next. That gives every release an author, a reviewer, a timestamp and a revert button, and the MR description is a natural place for release notes.

On GitLab Free, required approval rules are a paid feature, so the gate is enforced with protection settings that Free does include. In the GitOps repo, main is protected with "Allowed to push" set to no one and "Allowed to merge" set to Maintainers. In the application repo, v* tags are protected so only Maintainers can create a release tag. It is a weaker control than mandatory approvals, and we accept it knowingly.

Navigating ecosystem churn

Three of the components a tutorial from 2024 would have recommended were gone or unavailable by the time we built this. Designing around churn turned out to be as much of the work as designing the system.

ingress-nginx retirement, and Gateway API as the way forward

Kubernetes SIG Network and the Security Response Committee announced that ingress-nginx maintenance would stop in March 2026, and the Steering Committee later warned that staying on it after retirement leaves clusters exposed. Swapping in another Ingress controller would have worked, but our templates leaned on nginx-specific annotations that would have needed translating anyway.

Gateway API fits per-MR environments better than Ingress did. A platform-owned Gateway holds the listener and certificate; each preview namespace contributes only an HTTPRoute for its hostname, and the Gateway's allowedRoutes decides which namespaces may attach. The split mirrors who owns what: the platform owns the front door, the MR owns its route.

Because certificates come from DNS-01, the move does not change how they are issued. It does change where they live: with a shared Gateway, per-MR certificates mean one listener per hostname, so the natural end state is a single wildcard certificate for *.preview.<domain> on one listener.

We chose Envoy Gateway as the implementation. Dev and prod moved to it in May 2026. Previews are the last to move, in November 2026, because they switch to the wildcard certificate in the same step.

GitLab Free and the missing Project Access Token

The pull request generator needs a token with read_api to list MRs. On GitLab.com, project access tokens require Premium or Ultimate; on self-managed GitLab they are available on any tier.

Deploy tokens cover repository and registry reads but not the API, so they solve half the problem. For the other half we use a personal access token on a dedicated bot user that is a member of this one project only, scoped to read_api, with an expiry date. It is the one static credential left in the system, and it can read MR metadata, nothing more.

Results

Environments are live about 4.5 minutes after an MR opens, and all of the roughly 340 created since February 2026 were cleaned up without leaving a namespace or DNS record behind.

The image build is the critical path: about 2.5 minutes with a warm layer cache. Because the webhook creates the namespace within seconds of the MR opening, the certificate (about 90 seconds, mostly Route53 propagation) is issued in parallel with the build and rarely adds to the total. What does add time is that the pod is scheduled before its image exists, so kubelet's image pull back-off can add up to two minutes after the push. Cold-cache builds and that back-off account for most of the p95.

Orphan checks are worth running on a schedule rather than once for a case study: list open MRs from the GitLab API, list preview-mr-* namespaces and challenge records, and alert on the difference.

Lessons and what's next

• Reconcile, don't script. Environments that exist because a list says so clean themselves up. Environments that exist because a job ran need another job to remove them, and that job will eventually not run.
• Ownership decides cleanup. The namespace was the one resource Argo CD did not own by default, and it was the one that would have leaked. Check what your controller actually tracks before trusting cascade deletion.
• Federation works off EKS too. IRSA is just OIDC. Publishing the issuer and registering it in IAM removed the most sensitive secret the cluster would otherwise hold.
• Budget for churn. Two foundational tools changed status during this project. Choosing components with active upstreams and keeping vendor-specific configuration thin made each change a contained swap.

Next:

• Finish the Gateway API migration and move previews to a wildcard certificate.
• Run the orphan check as a scheduled job that posts to Mattermost.
• Add a TTL for MRs left open but idle, so a forgotten draft does not hold resources indefinitely.

Sources

• Argo CD pull request generator
• Ingress NGINX retirement announcement and Steering and Security Response Committees statement
• GoogleContainerTools/kaniko (archived)
• GitLab project access tokens
• cert-manager Route53 DNS-01
• Route53 IAM condition keys
• Let's Encrypt rate limits