# AWS multi-region deployment

This guide builds a two-region [region group](https://docs.diagrid.io/operate/platform-operations/multi-region) on AWS, with one global load balancer in front of both regions. It uses the same Terraform as the [AWS deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-installation-guide) guide.

Read [Multi-region high availability](https://docs.diagrid.io/operate/platform-operations/multi-region) first. It explains what a group is, what its members must share, and why you promote the database yourself. This page shows how to build it on AWS, with RDS cross-region read replicas, a KMS multi-region key, and AWS Global Accelerator as the global load balancer.

After you build the group, use [AWS failover and failback](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-failover) to run it.

## Architecture

```mermaid
---
title: Two AWS regions in one Catalyst region group
---
flowchart TD
  DNS("Route 53<br/>*.catalyst.example.com")
  GA("AWS Global Accelerator<br/>two static anycast addresses")

  DNS-->GA

  subgraph Active["AWS region 1 — active"]
    NLB1("Gateway load balancer<br/>health check: /diagrid/region/writable")
    EKS1("EKS — Catalyst data plane")
    RDS1[("RDS PostgreSQL<br/>primary")]
  end

  subgraph Passive["AWS region 2 — passive"]
    NLB2("Gateway load balancer<br/>health check: /diagrid/region/writable")
    EKS2("EKS — Catalyst data plane")
    RDS2[("RDS PostgreSQL<br/>read replica")]
  end

  GA=="targets healthy (200)"==>NLB1
  GA-."targets unhealthy (503)".->NLB2
  NLB1-->EKS1
  NLB2-->EKS2
  EKS1-->RDS1
  EKS2-->RDS2
  RDS1=="cross-region replication"==>RDS2
```

Clients reach the group through the accelerator's two static IP addresses. These addresses never change, so a failover moves traffic without clients looking up a new address or waiting for a DNS TTL. This matters because Dapr sidecars and the SDKs keep long-lived gRPC connections open, and those connections don't notice a DNS change.

The accelerator sends traffic based on the health of each region's gateway load balancer. That health check asks the region whether its database accepts writes. So the region running against the primary database gets the traffic, and promoting the replica is what moves it.

It's up to you to make sure only one region holds the primary at a time. Nothing in this setup enforces it.

Both regions run a full Catalyst data plane. A passive region isn't smaller: size both the same, and budget for two.

## Requirements checklist

The [region group requirements](https://docs.diagrid.io/operate/platform-operations/multi-region#requirements) list six settings the members must share. Five are always required, and one is optional. The control plane doesn't enforce any of them, so check all six before you create the group. This guide covers them like this:

| Requirement | How this guide covers it | Where you set it |
|---|---|---|
| One replicated database | An RDS PostgreSQL instance in the first region, and a cross-region read replica of it in the second | `postgresql_replicate_source_db_arn` (Terraform) |
| The same database name and admin user | The same values in both regions. If they differ, promoting the database replaces it instead. | `postgresql_db_name`, `postgresql_username` (Terraform) |
| The PostgreSQL secrets provider | Secrets are stored in the replicated database, not in one cluster's Kubernetes | `global.secrets.provider` (Helm) |
| The same key encryption key | An AWS KMS multi-region key: a primary in the first region and a replica in the second. AWS treats them as one key. | `kek_kms_enabled` (Terraform), `global.secrets.postgresql.kek_provider` and `aws_kms_key_id` (Helm) |
| The PostgreSQL scheduler backend | Scheduler jobs and actor reminders are stored in the replicated database, so they fail over with it and there's only one writer to promote | `scheduler_postgresql_instances = []` (Terraform), `agent.config.internal_dapr.scheduler.postgresql.use_global` (Helm) |
| One Dapr PKI root (optional) | A root you create once for the group, with an intermediate for each region | `agent.config.internal_dapr.pki` (Helm) |

The control plane warns you when two members aren't using the same key encryption key. It compares the value each region was *configured* with, not the key AWS resolves it to. If you point each region at the multi-region key by its own region-specific ARN, the control plane reports a mismatch even though it's the same key. That's why this guide configures both regions with the key ID instead.

## Before you start

Read the [AWS deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-installation-guide) guide and deploy a single region with it at least once. This page assumes you know those steps and only covers what's different.

You need:

If you run out of Elastic IPs, the error is easy to miss. Terraform allocates the addresses late in the apply, after it creates the cross-region database replicas. The `AddressLimitExceeded` errors scroll past, and the error you're left with is an EKS node group that never reaches `ACTIVE`, with `Instances failed to join the kubernetes cluster`. The real cause is that the NAT gateways never got addresses, so the private subnets had no route out. Check the quota instead of debugging the node group.

The Terraform is the same one the AWS deployment guide uses. It's in the [`guides/aws`](https://github.com/diagridio/charts/tree/main/guides/aws) directory of the [`diagridio/charts`](https://github.com/diagridio/charts) repository. Every variable this page adds defaults to single-region behavior, so nothing here changes what an existing deployment builds.

Clone the repository once for each region:

```bash
git clone https://github.com/diagridio/charts.git catalyst-west
git clone https://github.com/diagridio/charts.git catalyst-east
```

`catalyst-west/guides/aws` and `catalyst-east/guides/aws` each hold one region's Terraform state. You also need a third state for the group's shared entry point, in `guides/aws/terraform/region-group`. You can keep it in either clone, as long as you always apply it from the same one.

## 1. Decide the shared values

You decide every value in this step once for the whole group, and both regions use the same values. Most of them can't be changed later without rebuilding something. Decide all of them before you apply anything: the values below, the key encryption key, and whether you need a shared PKI root.

```bash
# The wildcard domain both regions serve. One domain, one certificate, and one
# set of project hostnames, so your applications can't tell there are two regions.
export INGRESS_DOMAIN="catalyst.example.com"

# The database name and admin user. Use the same values in both regions. If
# they differ, promoting the database replaces it instead.
export PG_DB_NAME="catalyst"
export PG_USERNAME="postgres"

# The admin password. RDS can't create a read replica of an instance whose
# password it manages, so you set this yourself. Don't put it in
# terraform.tfvars.
export TF_VAR_postgresql_password="<choose one>"
```

### The key encryption key

Catalyst encrypts every secret it stores in two layers. Each secret is encrypted with a data encryption key, and that key is encrypted with a **key encryption key (KEK)**. The encrypted rows replicate between the two regions, so if the regions use different KEKs, the second region can read the rows but can't decrypt them.

The Terraform creates the KEK as an **AWS KMS multi-region key**. The first region creates the primary key, and the second region creates a replica of it. A replica is the same key with its own ARN in its own region. Each region encrypts with its local copy, and the key material never gets copied into a cluster. You turn this on with `kek_kms_enabled` in steps 3 and 5. There's nothing else to decide here.

:::danger You can't recover a lost KEK

Every secret Catalyst stores for a project in this group is encrypted with this key. If you lose the key, nobody can read those secrets in either region. If you delete it in one region, that region can't read them anymore. That's why the Terraform gives the key a 30-day deletion window.

Rotate the key in both regions together. If you rotate it in only one, the other region can't read the secrets encrypted with the new key.

:::

If the key can't live in AWS, you can use the provider's **local KEK** instead. It's 32 random bytes, written as 64 hex characters, that you generate once and copy into both clusters:

```bash
export CATALYST_KEK=$(openssl rand -hex 32)
```

This works, but you lose what KMS gives you. You hold the key yourself, you copy it into both clusters by hand, and nothing checks that the two copies match. To use it, set `kek_kms_enabled = false` and use `kek_provider: local` in step 4.

### Optional: one shared PKI root

You only need this if a project in this group will set `enableWorkflowHistorySigning`. Without a shared root, a workflow that starts in one region and resumes in the other fails signature verification. Catalyst then marks it as tampered and stops it instead of retrying it.

Decide now, because you can't add this later. The first time a region joins, it records its Dapr trust anchors, and the control plane rejects any later join with different anchors. To add a shared root to a region that has already joined, Diagrid has to rejoin it for you. If you skip this step, each region creates its own root, which is fine as long as nothing in the group signs its workflow history.

Create one root for the group and one intermediate for each region, before either cluster exists. Dapr Sentry requires ECDSA P-256 keys:

```bash
# Once for the group, on a machine you trust. root.key never goes on a cluster.
openssl ecparam -name prime256v1 -genkey -noout -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/CN=catalyst-group" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,digitalSignature,keyCertSign,cRLSign" \
  -out root.crt

# Once per region.
for r in catalyst-west catalyst-east; do
  openssl ecparam -name prime256v1 -genkey -noout -out $r.key
  openssl req -new -key $r.key -subj "/CN=$r" -out $r.csr
  openssl x509 -req -in $r.csr -CA root.crt -CAkey root.key -CAcreateserial \
    -days 1825 -sha256 \
    -extfile <(printf "basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,digitalSignature,keyCertSign,cRLSign\n") \
    -out $r.crt
  cat $r.crt root.crt > $r-chain.crt
  openssl verify -CAfile root.crt $r.crt   # must print: OK
done
```

Both regions use the same root. Each region installs its own intermediate, in steps 4 and 7.

You can rotate a region's intermediate later without affecting anything else. The root stays the same, so the other region still trusts whatever this region issues next, and history signed with the old intermediate still verifies. To rotate the **root**, contact Diagrid.

## 2. Create both regions in Diagrid Cloud

Create two regions, and keep both join tokens. You group these names in [step 6](#6-create-the-region-group). Use the same names for the EKS clusters, because the entry point gives each cluster its own DNS name.

```bash
diagrid login

export JOIN_TOKEN_WEST=$(diagrid region create catalyst-west \
  --ingress placeholder.example.com | jq -r .joinToken)

export JOIN_TOKEN_EAST=$(diagrid region create catalyst-east \
  --ingress placeholder.example.com | jq -r .joinToken)
```

The placeholder is there because your domain isn't delegated yet. You set both regions to the real ingress domain in steps 4 and 7.

## 3. Deploy the first region's AWS infrastructure

In `catalyst-west/guides/aws`, create this `terraform/terraform.tfvars`:

```hcl
cluster_name = "catalyst-west"

postgresql_db_name = "catalyst"
postgresql_username = "postgres"

# RDS can't create a read replica of an instance whose password it manages,
# so a region group can't use an RDS-managed admin password.
postgresql_manage_master_user_password = false

# Both regions serve the same wildcard domain, so neither region owns its DNS
# record. The shared entry point owns it.
region_group_member = true

# Create the multi-region KMS key that encrypts the group's secrets, and the
# role the Catalyst pods use to access it. This region creates the primary key,
# and the second region replicates it.
kek_kms_enabled = true

# Don't create a separate database for the Dapr scheduler. Group members keep
# scheduler state in the shared database, so jobs and actor reminders replicate
# with everything else and there's only one writer to promote.
scheduler_postgresql_instances = []

# Failing back rebuilds this region's database as a replica of the other one,
# and RDS won't delete a protected instance. With deletion protection on, this
# region can't fail back.
postgresql_deletion_protection = false
```

Keep `postgresql_backup_retention_period` above zero. The default of 7 is fine. RDS can't replicate an instance that has automatic backups turned off.

Leave `postgresql_version` at its default, or set it to 16 or later. The passive region's scheduler reads changes from a read replica using logical decoding, and PostgreSQL only supports that on a replica from version 16. On an older version, the passive region's scheduler never starts. The Terraform refuses to plan a group with an older version.

**Set `postgresql_deletion_protection = false` now, before the first apply.** Turning it off later, when you rebuild a region as a replica, doesn't work in one step. Pointing an instance at a replication source replaces the instance. Terraform tries to delete it before it applies the new setting, and RDS refuses every time you retry. You'd have to clear the flag in a separate apply first, in the middle of a failover, with the group already running on one database.

Pass the AWS region and the domain on the `make` command line. `make` passes them as `-var`, so they override anything in `terraform.tfvars`:

```bash
make init
make plan  REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
```

Then follow steps 3 to 7 of the [AWS deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-installation-guide) guide as written: connect to the bastion host, configure `kubectl`, install the AWS Load Balancer Controller, install cert-manager and its Let's Encrypt ClusterIssuer, and install the monitoring tools.

Save these outputs for later:

```bash
make output postgresql_endpoint
make output postgresql_arn
make output kek_kms_key_id
make output kek_kms_key_arn
make output kek_kms_role_arn
```

`kek_kms_key_id` is the same in both regions, because a multi-region key and its replicas share one key ID. You configure both regions with it. `kek_kms_key_arn` is specific to this region, and the second region uses it to create the replica.

## 4. Install Catalyst in the first region

This is where you apply the shared settings from step 1. The values below are the single-region guide's values, plus everything a region group needs. The four blocks marked `SHARED` must be exactly the same in [step 7](#7-install-catalyst-in-the-second-region).

### Optional: this region's PKI intermediate

You only need this if you created a shared root in [step 1](#optional-one-shared-pki-root). It must be in the cluster before you install Catalyst, because the agent reads it when it starts, and you can't add it after the region has joined.

The Helm install below creates the `cra-agent` namespace, but these two objects need to exist before that. Create the namespace yourself first:

```bash
kubectl create namespace cra-agent

kubectl -n cra-agent create secret generic catalyst-group-issuer \
  --from-file=tls.crt=catalyst-west-chain.crt \
  --from-file=tls.key=catalyst-west.key
kubectl -n cra-agent create configmap catalyst-group-trust-anchors \
  --from-file=ca.crt=root.crt
```

Then add this block to the values file below, next to the existing `internal_dapr` block. It's the same in both regions: the names match, and only the contents of the secret differ.

```yaml
agent:
  config:
    internal_dapr:
      pki:
        issuer:
          secret:
            name: catalyst-group-issuer
            namespace: cra-agent
        trust:
          config_map:
            name: catalyst-group-trust-anchors
            namespace: cra-agent
```

### The values file and the Helm install

```bash
export RDS_POSTGRESQL_ENDPOINT="<value-from-postgresql_endpoint-output>"
export RDS_POSTGRESQL_PASSWORD="<the password from step 1>"
export KEK_KEY_ID="<value-from-kek_kms_key_id-output>"
export KEK_ROLE_ARN="<value-from-kek_kms_role_arn-output>"
export JOIN_TOKEN="<value-of-JOIN_TOKEN_WEST>"

cat > catalyst-west-values.yaml << EOF
global:
  serviceAccount:
    annotations:
      # How the agent and the management service get access to KMS. They have
      # no access key, so the AWS SDK falls back to its credential chain, which
      # on EKS uses this role.
      eks.amazonaws.com/role-arn: "$KEK_ROLE_ARN"

  # SHARED: store secrets in the replicated database, not in this cluster's
  # Kubernetes, so the other region can read them after a promotion.
  secrets:
    provider: postgresql
    postgresql:
      connection_string: "postgres://postgres:$RDS_POSTGRESQL_PASSWORD@$RDS_POSTGRESQL_ENDPOINT:5432/catalyst"
      # SHARED: the key ID, exactly. A multi-region key and its replica share
      # one key ID. The control plane compares the value each region was
      # configured with, so region-specific ARNs look like two different keys.
      kek_provider: awskms
      aws_kms_key_id: "$KEK_KEY_ID"
      # NOT SHARED: each region uses its local copy of the key.
      aws_region: us-west-2
      primary_key_version: 1

agent:
  config:
    project:
      # SHARED: the replicated database. The host is this region's own
      # instance, and the user is the same in both regions. The database name
      # isn't set here. It comes from postgresql_db_name in Terraform.
      default_managed_state_store_type: postgresql-shared-external
      external_postgresql:
        enabled: true
        auth_type: connectionString
        namespace: postgresql
        connection_string_host: $RDS_POSTGRESQL_ENDPOINT
        connection_string_port: 5432
        connection_string_username: postgres
        connection_string_password: "$RDS_POSTGRESQL_PASSWORD"
    internal_dapr:
      scheduler:
        # SHARED: store jobs and actor reminders in the replicated database, so
        # they survive a failover. use_global keeps them in the database above
        # instead of a separate instance. On etcd, or on any database that
        # isn't replicated, they're lost when you fail over.
        backend_type: postgresql
        postgresql:
          use_global: true

gateway:
  tls:
    enabled: true
    secretName: "cert-wildcard"
  envoy:
    service:
      type: LoadBalancer
      httpsPort: 443
      httpsTargetPort: 8443
      # Fixed, so the health check below can use a port that doesn't change.
      httpsNodePort: 30443
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
        # This is how failover works. The targets are healthy while this
        # region's database accepts writes, and the accelerator sends traffic
        # to the region whose load balancer has healthy targets.
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "HTTPS"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/diagrid/region/writable"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "30443"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-success-codes: "200"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
EOF

helm install catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent \
     --create-namespace \
     -f catalyst-west-values.yaml \
     --set join_token="${JOIN_TOKEN}"
```

The example puts the database password in the values file so you can see all the shared settings in one place. In production, put the secrets provider's connection string in a Kubernetes Secret and set `global.secrets.postgresql.existingSecret`, as described in the [Helm chart reference](./reference.md#postgresql-secrets-provider). The KMS key ID isn't a secret, because the role controls access to the key, not the ID. The database password is a secret.

**Why the health check port is fixed.** The gateway Service has two ports, and an AWS load balancer health check applies to every target group the Service creates. With `traffic-port`, the health check would also send an HTTPS request to the plain HTTP port, which fails. A target group with no healthy targets makes the accelerator treat the whole region as unhealthy. Fixing `httpsNodePort` gives the health check one port to use, so both target groups give the same answer: whether this region accepts writes. Any node port from 30000 to 32767 works, as long as each region uses it consistently.

**The endpoint accepts any host name.** A target group health check sends the target's own address as the `Host` header, and you can't change it. So the health check endpoint accepts any host name, and you don't need an extra DNS name or certificate for it. It also has no authentication, because a health check can't send credentials. The only thing it reveals is which region accepts writes.

Now finish setting up the region's ingress, following step 9 of the [AWS deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-installation-guide) guide:

```bash
# Terraform creates the hosted zone after the gateway load balancer exists,
# which is why this apply runs after the Helm install.
make apply REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

make output route53_zone_name_servers
make output route53_zone_id
make output region_ingress_endpoint
make output gateway_nlb_arn

diagrid region update catalyst-west \
  --ingress "<value-from-region_ingress_endpoint-output>"
```

These four outputs only exist once the gateway's load balancer does, which is why you read them here and not in step 3. If you ask for them before this apply, `make output` prints a Terraform error instead of an empty line. That means the output doesn't exist yet, not that the apply failed.

At your registrar, delegate the domain to the name servers in `route53_zone_name_servers`. Wait for the delegation to take effect, then issue the wildcard certificate the same way as step 9 of the single-region guide. The gateway pod isn't ready until the certificate exists.

Save `route53_zone_id` and `gateway_nlb_arn`. The second region uses this hosted zone instead of creating another one for the same domain, and the entry point stack writes the wildcard record here. The load balancer ARN lets you apply that stack later without contacting this region. See [step 8](#8-create-the-shared-entry-point).

## 5. Deploy the second region's AWS infrastructure

In `catalyst-east/guides/aws`, create this `terraform/terraform.tfvars`:

```hcl
cluster_name = "catalyst-east"

# The same as region 1. If they differ, promoting the database replaces it.
postgresql_db_name = "catalyst"
postgresql_username = "postgres"

postgresql_manage_master_user_password = false

region_group_member = true

# Use region 1's hosted zone instead of creating a second zone for the same
# domain.
route53_zone_id = "Z0123456789ABCDEFGHIJ"

# Replicate region 1's KEK instead of creating a new key. A new key would be a
# different key, and each region could only read the secrets it stored itself.
kek_kms_enabled                = true
kek_kms_replica_source_key_arn = "arn:aws:kms:us-west-2:111122223333:key/mrk-abcdef"

# Build this region's database as a cross-region read replica of region 1's.
postgresql_replicate_source_db_arn = "arn:aws:rds:us-west-2:...:db:catalyst-west-postgresql"

# The same as region 1, for the same reasons.
scheduler_postgresql_instances = []
postgresql_deletion_protection = false
```

```bash
make init
make plan  REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
```

`REGION_INGRESS_ENDPOINT` is the same as in region 1. Only `REGION` is different. If you leave `REGION` out, `make` builds this region in `us-west-2`, next to the first one, whatever the working directory is called.

A replica gets its admin password from its source through replication, so `TF_VAR_postgresql_password` has no effect while the database is a replica. Set it to the same value anyway, because it takes effect after a promotion.

Then prepare the cluster the same way as region 1: bastion, `kubectl`, AWS Load Balancer Controller, cert-manager and its ClusterIssuer, and monitoring. This region's cert-manager writes to the hosted zone that region 1 created. The IAM policy from the Terraform isn't limited to one zone, so you don't need to change anything.

## 6. Create the region group

Create the group now, while both regions are registered and still empty, and before you install Catalyst in the second region:

```bash
diagrid region group create my-group --regions catalyst-west,catalyst-east
```

The group only needs the two regions you created in [step 2](#2-create-both-regions-in-diagrid-cloud). A member doesn't need to be online, and the second region doesn't need Catalyst installed yet.

**Why the group comes first.** The second region's database is a read replica from the start, so its agent must set up without writing anything. The agent decides how to set up based on the group membership the control plane sends it **when it joins**. If the region joins before the group exists, the agent thinks it's a standalone region, tries to create tables in a read-only database, and exits:

```
level=fatal msg="error creating region mcp store: ... cannot execute CREATE TABLE
in a read-only transaction (SQLSTATE 25006)"
```

It fixes itself once you create the group, because the next restart picks up the membership. But if you create the group first, the region starts as a member from the beginning and never gets into a crash loop.

## 7. Install Catalyst in the second region

Create `catalyst-east-values.yaml` from the same heredoc as `catalyst-west-values.yaml` in step 4, with only these changes:

- Set `RDS_POSTGRESQL_ENDPOINT` to **this** region's instance. Each region connects to the database instance in its own AWS region. Catalyst doesn't use a shared writer or reader endpoint.
- Set `aws_region` to `us-east-1`, and `KEK_ROLE_ARN` to this region's own role. Each region uses its local copy of the key through a role in its own region.
- Set `JOIN_TOKEN` to `JOIN_TOKEN_EAST`.
- You can keep `httpsNodePort` at `30443`.

If you created a shared root in [step 1](#optional-one-shared-pki-root), install this region's intermediate first, the same way as in [step 4](#optional-this-regions-pki-intermediate), but with `catalyst-east-chain.crt` and `catalyst-east.key`. The secret and ConfigMap have the same names in both regions, so the values block stays the same.

**Don't change `aws_kms_key_id`.** The replica shares the primary key's ID, and both regions must use that ID. Before you install, run `make output kek_kms_key_id` in both working directories and check that they print the same value.

Everything marked `SHARED` stays exactly the same: the same user, the same key ID and key version, and the same scheduler backend. The database name must match too, but you set that in Terraform with `postgresql_db_name`, not in these values.

Compare the two values files before you install. Each cluster's API server is private, so you run `kubectl` and `helm` on each region's own bastion. The two bastions are in different VPCs and different AWS regions, so neither one has both files. Write both files on your own machine, compare them there, and then copy each one to its bastion. The `mask` function hides the three values that are allowed to differ (the database endpoint, the AWS region, and the role ARN), so any output left over is a mistake:

```bash
mask() {
  sed -E -e 's/[a-z0-9.-]+\.rds\.amazonaws\.com/RDS_ENDPOINT/g' \
         -e 's/(us|eu|ap)-[a-z]+-[0-9]/AWS_REGION/g' \
         -e 's#arn:aws:iam::[0-9]+:role/[A-Za-z0-9_+=,.@-]+#KEK_ROLE#g' "$1"
}
diff <(mask catalyst-west-values.yaml) <(mask catalyst-east-values.yaml)
```

The diff should print nothing.

On this region's bastion, install Catalyst with the same command as step 4, using this region's values file and join token:

```bash
export JOIN_TOKEN="<value-of-JOIN_TOKEN_EAST>"

helm install catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent \
     --create-namespace \
     -f catalyst-east-values.yaml \
     --set join_token="${JOIN_TOKEN}"
```

Then finish this region's ingress, as you did for region 1. This region doesn't create a hosted zone, because it uses region 1's. But you still run the apply after the Helm install, because the region's DNS record and the `gateway_nlb_arn` output both need the gateway load balancer, which the install creates.

```bash
make apply REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

make output region_ingress_endpoint
make output gateway_nlb_arn

diagrid region update catalyst-east \
  --ingress "<value-from-region_ingress_endpoint-output>"
```

This apply also creates `catalyst-east.catalyst.example.com`, a name that reaches only this region. Each region creates its own. A group member needs one because the wildcard name points to the shared entry point, not to the region. Save this region's `gateway_nlb_arn` too.

Both regions now use the same ingress domain. Issue this region's wildcard certificate the same way, from the shared hosted zone.

If you created a shared root in [step 1](#optional-one-shared-pki-root), check that the root fingerprint is the same in both clusters:

```bash
kubectl -n root-dapr-system get configmap dapr-trust-bundle \
  -o jsonpath='{.data.ca\.crt}' | openssl x509 -noout -fingerprint -sha256
```

## 8. Create the shared entry point

Both regions are now running, each with its own gateway load balancer, but nothing resolves `*.catalyst.example.com` yet. The entry point is a third Terraform state. It holds one Global Accelerator in front of both load balancers, and the wildcard DNS record that points to it.

In `catalyst-west/guides/aws` (pick one clone and always use the same one), create `terraform/region-group/terraform.tfvars`:

```hcl
region_ingress_endpoint = "catalyst.example.com"
route53_zone_id         = "Z0123456789ABCDEFGHIJ"

primary_aws_region   = "us-west-2"
primary_cluster_name = "catalyst-west"

secondary_aws_region   = "us-east-1"
secondary_cluster_name = "catalyst-east"

# Each region's gateway_nlb_arn output. With both set, this stack doesn't need
# to contact either region, so you can still apply it while one region is
# down. That's exactly when you need to change its traffic dial. If you leave
# them out, the stack finds each load balancer by tag, which only works while
# that region responds.
primary_gateway_lb_arn   = "arn:aws:elasticloadbalancing:us-west-2:111122223333:loadbalancer/net/k8s-craagent-gateway/abc123"
secondary_gateway_lb_arn = "arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/net/k8s-craagent-gateway/def456"
```

```bash
make group-init
make group-plan
make group-apply

make group-output accelerator_ip_addresses
make group-output region_endpoints
```

This creates:

- **One accelerator** with two static anycast addresses. It forwards TCP port 443 to both regions, and nothing else. The gateway handles TLS itself, so the accelerator passes connections through without decrypting them. `accelerator_ip_addresses` lists the addresses to allow in your firewall. They stay the same through every failover. If your gateway uses a different port, change `listener_port`.
- **`*.catalyst.example.com`**, pointing to the accelerator. Every project hostname and region service name under the domain now resolves to it.

It **doesn't** create the per-region names. Each region already created its own name (`catalyst-west.catalyst.example.com` and `catalyst-east.catalyst.example.com`) in its earlier apply. These names point straight to the region's load balancer and skip the accelerator. They work whether or not the region is getting traffic, so you can use them to check a passive region. They live in each region's own state, so this stack needs nothing from a region except its load balancer ARN.

Check that each region answers for itself:

```bash
for r in catalyst-west catalyst-east; do
  printf '%s: ' "$r"
  curl -so /dev/null -w '%{http_code}\n' "https://$r.$INGRESS_DOMAIN/diagrid/region/writable"
done
```

Region 1 returns `200` and region 2 returns `503`. Both are group members, region 1 has the database that accepts writes, and the passive region reports that it shouldn't get traffic. [Step 10](#10-verify-the-group) checks this against what the control plane reports.

If either region returns `404`, it isn't a group member. Check that it's in the group before you continue. If you get `000` or a TLS error, the wildcard certificate hasn't been issued yet, or the DNS name hasn't propagated.

## 9. Create a project

Only create projects after the group exists:

```bash
diagrid project create my-app --region-group my-group

diagrid project get my-app --output json | jq '.status.instances'
```

Each entry shows a region the project runs in and whether it's ready. Use `jq '.status.regionEndpoints'` to see the address for each region. A project in the group runs in each region and has one set of hostnames: `http-prj<id>.catalyst.example.com` and the matching `grpc-` name. They resolve through the accelerator to whichever region is active. Your applications use one endpoint and don't know when a failover happens.

If you create resources during a failover, they wait on the passive side until writes work again.

## 10. Verify the group

Run these four checks before the group handles anything important.

**1. The control plane reports no mismatches.** This is the first point where the control plane can compare the two members. It reports a mismatched KEK or an unshared PKI root here:

```bash
diagrid region group get my-group --output json | jq '.status.messages'
```

You want an empty list. If you see `region catalyst-east does not resolve the KEK of region catalyst-west`, the two regions are configured with different keys. Fix it now, before any project stores a secret. After that, each region holds rows the other can't read.

**2. Exactly one member accepts writes.**

```bash
diagrid region group get my-group --output json | jq '.status'
```

The status shows which member accepts writes. If a member reports nothing, that isn't the same as `false`. It means the member's agent has stopped sending heartbeats, which is a separate problem.

**3. Each region agrees with the control plane.** Run the loop from [step 8](#8-create-the-shared-entry-point) again. The region that check 2 showed as the writer returns `200`, and the other returns `503`. The passive region's load balancer targets become unhealthy within about 20 seconds, and the accelerator sends everything to the active region.

```bash
for r in catalyst-west catalyst-east; do
  printf '%s: ' "$r"
  curl -so /dev/null -w '%{http_code}\n' "https://$r.$INGRESS_DOMAIN/diagrid/region/writable"
done
```

**4. Replication lag reaches your monitoring.** Every passive member exports [`cra_region_replication_lag_seconds`](https://docs.diagrid.io/operate/platform-operations/multi-region/metrics). That metric is your [RPO](https://docs.diagrid.io/operate/platform-operations/multi-region#recovery-objectives-rpo-and-rto), so check that your monitoring collects it, and set an alert on it before you go live.

## 11. Rehearse a failover

Do this before the group serves production.

A full failover and failback with [AWS failover and failback](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-failover) rebuilds **both** regions' databases, so treat the rehearsal as planned maintenance. You can't undo it. Rehearse:

Time how long it takes from "we decided to promote" to "the first write succeeded in the new region". That's your real [RTO](https://docs.diagrid.io/operate/platform-operations/multi-region#recovery-objectives-rpo-and-rto), and most of it is decision time, not machine time.

## Tear the group down

For each region, follow the single-region guide's [teardown](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-installation-guide#tear-the-region-down). Run `helm uninstall` first, so the AWS Load Balancer Controller deletes the gateway's load balancer while it's still running. A group also needs things done in a specific order:

1. **Delete the projects, then the group.** Run `diagrid project delete`, then `diagrid region group delete my-group`. You can't delete a group that still has projects, and the error names the first one it finds. Deleting the group doesn't delete its regions. Once a region's last group project has drained, you can create projects in it again.
2. **Destroy the shared entry point.** In the working directory that holds its state, run `make group-destroy`. This removes the accelerator and the wildcard record. Every project hostname stops working at this point, so do it before you tear down either region.
3. **Destroy the replica's region before the region it replicates from**, so AWS doesn't promote an instance you're about to delete.
4. **Destroy the region that created the hosted zone last**, whichever region is active by then. Region 1 created the zone, and region 2 only added records to it. You can't delete the zone while region 2's record is still in it.

Steps 3 and 4 depend on what each region *is*, not on which one is active. If the group has failed over and not failed back, the roles are swapped: region 1 still owns the zone, but it's now the replica. Follow the steps in order and they still work.

Both regions need `postgresql_deletion_protection = false`, which the tfvars in [step 3](#3-deploy-the-first-regions-aws-infrastructure) and [step 5](#5-deploy-the-second-regions-aws-infrastructure) already set. Each region takes a final snapshot of its database when you destroy it, unless you set `postgresql_skip_final_snapshot`.

The group's KMS key is a multi-region key. When you destroy it, AWS schedules the primary and its replica for deletion after a 30-day waiting period instead of deleting them right away. That's intended: it means you can recover from an accidental destroy.

## Limitations

These limits come from this Terraform, on top of the [limitations of a region group](https://docs.diagrid.io/operate/platform-operations/multi-region#limitations):

- It doesn't promote anything by itself. The only thing that reacts when a region stops accepting writes is the load balancer health check, and that only moves traffic.
- It doesn't demote anything. RDS can't turn a primary back into a replica, so the only way to demote a region is to destroy its databases and rebuild them as replicas of the new primary.
- It doesn't cap replication lag, but the Catalyst agent does measure it.
- It doesn't fail over managed Kafka pub/sub or Redis rate-limit counters, which are regional. A project in a group needs its own pub/sub on a multi-region broker.
- It can't size the two regions differently. Both regions use the same sizing variables, because a passive region is a full region.

## Next steps

- [AWS failover and failback](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-failover) — The runbook: promote a region, move traffic, and rebuild the old one as a replica.
- [Multi-region high availability](https://docs.diagrid.io/operate/platform-operations/multi-region) — What a region group is, what its members share, and how a failover works on any cloud.
- [Production planning](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/production-planning) — Size, secure, and operate a self-managed region before it takes production traffic.