AWS multi-region deployment

This guide builds a two-region region group on AWS, from the same Terraform as the AWS deployment guide, and places a single global load balancer in front of both regions.

Read Multi-region high availability first. It covers what a group is, what its members must share, and why promotion is a decision you make rather than one the platform makes. This page is the AWS realization of it: RDS cross-region read replicas, a KMS multi-region key, and AWS Global Accelerator as the global load balancer.

Once the group is built, AWS failover and failback is the runbook you operate it with.

Architecture

Every client reaches the group at the accelerator's two static addresses. Those addresses never change, so a failover moves traffic without any client re-resolving a name or waiting for a DNS TTL to expire. That matters here, because Dapr sidecars and the SDKs hold long-lived gRPC connections that a DNS change would not disturb.

The accelerator routes traffic according to each region's gateway load balancer health, and that health check asks the region whether its database accepts writes. The region running against the primary database is therefore the region that receives traffic, and promoting the replica is what moves it.

Ensuring that exactly one region holds the primary at any time is your responsibility. Nothing in the design enforces it.

Both regions run a full Catalyst data plane. A passive region is not a reduced region: size both identically, and budget for two.

Requirements checklist

The region group requirements list six settings the members must share, five of them unconditional, and explain what each one prevents. The control plane enforces none of them, so verify all six before creating the group. This guide satisfies them as follows:

Requirement	How this guide provides it	Where you set it
One replicated database	An RDS PostgreSQL instance in the first region, and a cross-region read replica of it in the second	`postgresql_replicate_source_db_arn` (Terraform)
The same database name and admin user	The same literal values in both regions, because a mismatch turns a promotion into a database replacement	`postgresql_db_name`, `postgresql_username` (Terraform)
The PostgreSQL secrets provider	Secrets live in the replicated cluster instead of in one cluster's Kubernetes	`global.secrets.provider` (Helm)
The same key encryption key	An AWS KMS multi-region key — a primary in the first region and a replica of it in the second, which is one key identity	`kek_kms_enabled` (Terraform), `global.secrets.postgresql.kek_provider` and `aws_kms_key_id` (Helm)
The PostgreSQL scheduler backend	Scheduler instances replicated the same way as the project database	`scheduler_postgresql_replicate_source_db_arns` (Terraform), `agent.config.internal_dapr.scheduler` (Helm)
One Dapr PKI root — optional	A root you create once for the group, with an intermediate per region	`agent.config.internal_dapr.pki` (Helm)

The key encryption key comparison reads the configured key, not the resolved one

The control plane warns when two members do not resolve the same key encryption key, and it compares the key each region was configured with. Two regions that name one multi-region key by their own region-local ARNs are using the same key and are still reported as a mismatch — which is why this guide configures both regions with the key id.

Before you start

Read the AWS deployment guide first and deploy a single region from it at least once. This page assumes those steps and only describes what differs.

You need:

Two AWS regions in one AWS account, and a wildcard domain you control.
Separate Terraform state per region. You apply the same stack twice, once per region, from two working directories. A single configuration holding both regions cannot produce a plan while one region's API is unreachable, and that is exactly when you need to promote.
Empty regions. A region that already holds a project cannot join a group, and a region in a group takes no standalone projects. Create both regions, group them, then create your projects.
A new project. You cannot move an existing single-region project into a group. That means new regions, a new project, new hostnames and API keys, and a redeploy of the applications that call it.
A decision about workflow history signing, made in step 1. It requires a shared PKI root installed in both regions before they first join, and it cannot be added afterwards without Diagrid rejoining the regions on your behalf.

Minimum chart version

The failover mechanism on this page is a load balancer health check reading /diagrid/region/writable. That endpoint is published by Catalyst chart 1.113.0 and later. On an earlier chart the endpoint answers 404 in every region, the load balancer has no healthy target anywhere, and the group cannot fail over.

The Terraform is the same one the AWS deployment guide uses, in the guides/aws directory of the diagridio/charts repository. Every variable this page introduces defaults to the single-region behavior, so an existing deployment plans unchanged.

Set up one working directory per region, plus one for the group's shared entry point:

git clone https://github.com/diagridio/charts.git catalyst-west
git clone https://github.com/diagridio/charts.git catalyst-east

catalyst-west/guides/aws and catalyst-east/guides/aws each hold one region's Terraform state. The entry point is a third state, in guides/aws/terraform/region-group under either clone; either clone is fine, provided the stack is always applied from the same one.

1. Decide the shared values

Every value in this step is decided once for the group and used identically by both regions, and most of them cannot be changed later without rebuilding something. Decide all of them — the values below, the key encryption key, and whether the group needs a shared PKI root — before applying anything.

# The wildcard domain both regions serve. One domain, one certificate, one
# set of project hostnames — the group is invisible to your applications.
export INGRESS_DOMAIN="catalyst.example.com"

# The database name and admin user. Identical in both regions: promotion
# reconciles them against what the promoted instance already reports, and a
# mismatch is a replacement rather than a promotion.
export PG_DB_NAME="catalyst"
export PG_USERNAME="postgres"

# The admin password. RDS cannot create a read replica of an instance whose
# credentials it manages, so you supply this rather than letting RDS generate
# it. Keep it out of terraform.tfvars.
export TF_VAR_postgresql_password="<choose one>"

The key encryption key

Catalyst envelope-encrypts every secret it stores: each secret is sealed with a data encryption key, and that key is sealed with a key encryption key (KEK). The sealed rows replicate between the two regions, so a region holding a different KEK reads them and cannot decrypt them.

The Terraform builds this as an AWS KMS multi-region key: the first region creates the primary, the second creates a replica of it. A replica is the same key identity with its own ARN in its own region, so each region encrypts against the copy next to it and no key material is ever copied into a cluster. Steps 3 and 5 enable it with kek_kms_enabled, and there is nothing further to decide here.

The KEK has no recovery path

Every secret Catalyst stores for a project in this group is sealed with this key. If the key is lost, the rows are unreadable in both regions. If it is deleted in one region, that region can no longer read them. The Terraform gives it a 30-day deletion window for that reason.

Rotating it is a group-level operation rather than a per-region one. Rotate both members together, or one of them will no longer be able to read what the other wrote.

The alternative is the provider's local KEK: 32 random bytes as 64 hex characters, generated once and pasted into both clusters.

export CATALYST_KEK=$(openssl rand -hex 32)

This works, and it is the right choice if the key must not live in AWS. It gives up what KMS provides: the key exists as a value you hold, you copy it into both clusters yourself, and nothing verifies that the second copy matches the first. Set kek_kms_enabled = false and use kek_provider: local in step 4 if you take this path.

Optional: one shared PKI root

You need this only if a project in this group will set enableWorkflowHistorySigning. Without it, a workflow that starts in one region and resumes in the other fails signature verification, and a resumed workflow that fails verification is tombstoned as tampered rather than retried.

Decide this now, because you cannot decide it later. On its first join a region records the Dapr trust anchors it presents, and the control plane refuses any later join whose anchors differ. Installing a shared root on an already-joined region requires Diagrid to rejoin it on your behalf. If you skip this, each region generates a root of its own, which is correct when nothing in the group signs its workflow history.

Create one root for the group and one intermediate per region now, before either cluster exists. Dapr's Sentry requires ECDSA P-256 keys:

# Once for the group, on a machine you trust. root.key never reaches a cluster.
openssl ecparam -name prime256v1 -genkey -noout -out root.key
openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/CN=catalyst-group" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,digitalSignature,keyCertSign,cRLSign" \
  -out root.crt

# Once per region.
for r in catalyst-west catalyst-east; do
  openssl ecparam -name prime256v1 -genkey -noout -out $r.key
  openssl req -new -key $r.key -subj "/CN=$r" -out $r.csr
  openssl x509 -req -in $r.csr -CA root.crt -CAkey root.key -CAcreateserial \
    -days 1825 -sha256 \
    -extfile <(printf "basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,digitalSignature,keyCertSign,cRLSign\n") \
    -out $r.crt
  cat $r.crt root.crt > $r-chain.crt
  openssl verify -CAfile root.crt $r.crt   # must print: OK
done

The root is the same in both regions; each region installs the intermediate carrying its own name, in steps 4 and 6.

Rotating a region's intermediate later is routine and affects nothing else: the root does not change, so the other region already trusts whatever this region issues next, and history signed under the old intermediate continues to verify. Rotating the root requires Diagrid.

2. Create both regions in Diagrid Cloud

Two regions, two join tokens. The names used here are the names grouped in step 8. Use the same names for the EKS clusters, because the entry point gives each cluster a DNS name of its own.

diagrid login

export JOIN_TOKEN_WEST=$(diagrid region create catalyst-west \
  --ingress placeholder.example.com | jq -r .joinToken)

export JOIN_TOKEN_EAST=$(diagrid region create catalyst-east \
  --ingress placeholder.example.com | jq -r .joinToken)

Both regions receive the same real ingress domain in steps 4 and 6. The placeholder is used only because the domain is not delegated yet.

3. Deploy the first region's AWS infrastructure

From catalyst-west/guides/aws, with this terraform/terraform.tfvars:

cluster_name = "catalyst-west"

postgresql_db_name = "catalyst"
postgresql_username = "postgres"

# RDS refuses to create a read replica of a source whose credentials it
# manages, so a region group cannot use an RDS-managed admin password.
postgresql_manage_master_user_password = false

# Both regions of a group serve the same wildcard domain, so neither of them
# owns that domain's record: the shared entry point does.
region_group_member = true

# Create the multi-region KMS key the group seals its secrets with, and the
# role the Catalyst pods assume to use it. This region creates the primary;
# the second region replicates it.
kek_kms_enabled = true

Keep postgresql_backup_retention_period above zero; the default of 7 is fine. RDS cannot replicate an instance whose automatic backups are disabled.

The AWS region and the domain go on the make line, which passes both as -var and so overrides whatever terraform.tfvars says:

make init
make plan  REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

Then follow steps 3 to 7 of the AWS deployment guide unchanged: connect to the Bastion host, configure kubectl, install the AWS Load Balancer controller, install cert-manager and its Let's Encrypt ClusterIssuer, and install the monitoring tools.

Note the outputs you need later:

make output postgresql_endpoint
make output postgresql_arn
make output scheduler_postgresql_endpoints
make output scheduler_postgresql_arns
make output kek_kms_key_id
make output kek_kms_key_arn
make output kek_kms_role_arn

kek_kms_key_id is the same string in both regions, because a multi-region key and its replicas share one key id, and it is the value both regions are configured with. kek_kms_key_arn is region-local, and it is the value the second region replicates from.

4. Install Catalyst in the first region

This is where the shared configuration from step 1 is applied. The values below are the single-region guide's values plus everything a region group requires; the four blocks marked as shared must be identical in step 6.

Optional: this region's PKI intermediate

Required only if you created a shared root in step 1. It must exist in the cluster before Catalyst is installed: the agent reads it at startup, and a region that has already joined cannot be given one afterwards.

kubectl -n cra-agent create secret generic catalyst-group-issuer \
  --from-file=tls.crt=catalyst-west-chain.crt \
  --from-file=tls.key=catalyst-west.key
kubectl -n cra-agent create configmap catalyst-group-trust-anchors \
  --from-file=ca.crt=root.crt

Then add this to the values file below, alongside the internal_dapr block it already has. The block is identical in both regions: the names are the same, and only the contents of the secret differ.

agent:
  config:
    internal_dapr:
      pki:
        issuer:
          secret:
            name: catalyst-group-issuer
            namespace: cra-agent
        trust:
          config_map:
            name: catalyst-group-trust-anchors
            namespace: cra-agent

The values file and the Helm install

export RDS_POSTGRESQL_ENDPOINT="<value-from-postgresql_endpoint-output>"
export RDS_SCHEDULER_ENDPOINT="<pg1 value-from-scheduler_postgresql_endpoints-output>"
export RDS_POSTGRESQL_PASSWORD="<the password from step 1>"
export KEK_KEY_ID="<value-from-kek_kms_key_id-output>"
export KEK_ROLE_ARN="<value-from-kek_kms_role_arn-output>"
export JOIN_TOKEN="<value-of-JOIN_TOKEN_WEST>"

cat > catalyst-values.yaml << EOF
global:
  serviceAccount:
    annotations:
      # How the agent and the management service reach KMS. They are given no
      # access key, so the AWS SDK falls through to the credential chain, which
      # on EKS is this role.
      eks.amazonaws.com/role-arn: "$KEK_ROLE_ARN"

  # SHARED: secrets live in the replicated cluster, not in this cluster's
  # Kubernetes, so the other region can read them after a promotion.
  secrets:
    provider: postgresql
    postgresql:
      connection_string: "postgres://postgres:$RDS_POSTGRESQL_PASSWORD@$RDS_POSTGRESQL_ENDPOINT:5432/catalyst"
      # SHARED: the key id, byte for byte. A multi-region key and its replica
      # share one key id, and the control plane compares what each region was
      # configured with rather than what it resolves to — so naming the key by
      # its region-local ARN in each region reads as two different keys.
      kek_provider: awskms
      aws_kms_key_id: "$KEK_KEY_ID"
      # NOT shared: each region uses the copy of the key next to it.
      aws_region: us-west-2
      primary_key_version: 1

agent:
  config:
    project:
      # SHARED: the replicated cluster. The host is this region's own
      # instance; the database and user are the same on both sides.
      default_managed_state_store_type: postgresql-shared-external
      external_postgresql:
        enabled: true
        auth_type: connectionString
        namespace: postgresql
        connection_string_host: $RDS_POSTGRESQL_ENDPOINT
        connection_string_port: 5432
        connection_string_username: postgres
        connection_string_password: "$RDS_POSTGRESQL_PASSWORD"
        connection_string_database: catalyst
    internal_dapr:
      scheduler:
        # SHARED: jobs and actor reminders on the replicated scheduler
        # instances. On etcd, or on anything not replicated, they are lost at
        # the moment of the failover.
        backend_type: postgresql
        postgresql:
          use_global: false
          connections:
            - host: $RDS_SCHEDULER_ENDPOINT
              port: 5432
              username: postgres
              password: "$RDS_POSTGRESQL_PASSWORD"
              database: scheduler

gateway:
  tls:
    enabled: true
    secretName: "cert-wildcard"
  envoy:
    service:
      type: LoadBalancer
      httpsPort: 443
      httpsTargetPort: 8443
      # Pinned so the health check below can name a port that does not move.
      httpsNodePort: 30443
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
        # This is the failover mechanism. The targets are healthy while this
        # region's database accepts writes, and the accelerator sends traffic
        # to whichever region's load balancer reports healthy targets.
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "HTTPS"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/diagrid/region/writable"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "30443"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-success-codes: "200"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
EOF

helm install catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent \
     --create-namespace \
     -f catalyst-values.yaml \
     --set join_token="${JOIN_TOKEN}" \
     --version 1.113.0

Keep the database password out of the values file

The inline form above is shown so that the whole shared configuration is visible in one place. In production, put the secrets provider's connection string in a Kubernetes Secret instead and set global.secrets.postgresql.existingSecret, as the Helm chart reference describes. With the KMS KEK the key id is not a secret, because access is granted by the role rather than by the id, but the database password in that connection string is.

Why the health check port is pinned

The gateway Service publishes two ports, and an AWS load balancer health check applies to every target group the Service produces. traffic-port would send an HTTPS probe to the plaintext port as well, which fails, and a target group with no healthy targets causes the accelerator to treat the whole region as unhealthy. Pinning httpsNodePort gives the health check a single port to name, so both target groups report the same answer: whether this region accepts writes.

Any node port in the 30000–32767 range works, provided each region uses its own consistently.

The endpoint answers under any host name

A load balancer target group health check sends the target's own address as the Host header and cannot be given a custom one. The health check endpoint answers under any host name for that reason, so no extra DNS name or certificate is involved. It carries no authentication, because a health check cannot present a credential, and the only information it discloses is which of the two regions currently accepts writes.

Now finish the region's ingress, following step 9 of the AWS deployment guide:

# The hosted zone is created once the gateway load balancer exists, which is
# why this apply runs after the Helm install rather than before it.
make apply REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

make output route53_zone_name_servers
make output route53_zone_id
make output region_ingress_endpoint
make output gateway_nlb_arn

diagrid region update catalyst-west \
  --ingress "<value-from-region_ingress_endpoint-output>"

Delegate the domain to the nameservers in route53_zone_name_servers at your registrar, wait for the delegation to propagate, then issue the wildcard certificate exactly as the single-region guide's step 9 does. The gateway pod stays unready until that certificate exists.

Keep route53_zone_id and gateway_nlb_arn. The second region joins that zone rather than creating a second one for the same name, and the entry point stack writes the wildcard record there. The load balancer ARN is what allows that stack to be applied later without querying this region; see step 7.

5. Deploy the second region's AWS infrastructure

From catalyst-east/guides/aws, with this terraform/terraform.tfvars:

cluster_name = "catalyst-east"

# Identical to region 1. Promotion reconciles these against what the promoted
# instance already reports.
postgresql_db_name = "catalyst"
postgresql_username = "postgres"

postgresql_manage_master_user_password = false

region_group_member = true

# Join region 1's hosted zone instead of creating a second zone for the same
# name.
route53_zone_id = "Z0123456789ABCDEFGHIJ"

# Replicate region 1's KEK rather than creating a second key. A second key
# would be a second key identity, and the group's secrets would be readable in
# only one of the two regions.
kek_kms_enabled                = true
kek_kms_replica_source_key_arn = "arn:aws:kms:us-west-2:111122223333:key/mrk-abcdef"

# Build this region's databases as cross-region read replicas of region 1's.
postgresql_replicate_source_db_arn = "arn:aws:rds:us-west-2:...:db:catalyst-west-postgresql"

scheduler_postgresql_replicate_source_db_arns = {
  pg1 = "arn:aws:rds:us-west-2:...:db:catalyst-west-scheduler-pg1-postgresql"
}

make init
make plan  REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

REGION_INGRESS_ENDPOINT is identical to region 1's; REGION is the only value that differs. A make line that omits REGION builds this region in us-west-2 alongside the first one, regardless of the working directory's name.

A replica inherits its admin password from its source through replication, so TF_VAR_postgresql_password does not apply while it is a replica. Set it to the same value regardless: it takes effect after promotion.

Then run the same cluster preparation as region 1: Bastion, kubectl, AWS Load Balancer controller, cert-manager and its ClusterIssuer, monitoring. This region's cert-manager writes into the hosted zone the first region created; the IAM policy the Terraform attaches is not scoped to a single zone, so nothing extra is needed.

6. Install Catalyst in the second region

The same values file as step 4, with the following changes and nothing else:

RDS_POSTGRESQL_ENDPOINT and RDS_SCHEDULER_ENDPOINT are this region's instances. Each region connects to the instance in its own AWS region; Catalyst uses no shared writer or reader endpoint.
aws_region is us-east-1, and KEK_ROLE_ARN is this region's own role. Each region uses the copy of the key next to it, through a role in its own account region.
JOIN_TOKEN is JOIN_TOKEN_EAST.
httpsNodePort can remain 30443.

If you created a shared root in step 1, install this region's intermediate first, exactly as in step 4 but from catalyst-east-chain.crt and catalyst-east.key. The secret and configmap are named the same in both regions, so the values block itself does not change.

aws_kms_key_id does not change: the replica shares the primary key's id, and that shared id is what both regions must be configured with. Confirm it before installing — make output kek_kms_key_id in both working directories must print the same string.

Everything marked SHARED remains identical, character for character: the same database name, the same user, the same key id and key version, and the same scheduler backend.

Compare the two values files before installing, on whichever machine holds both. The mask below removes the two values the regions are allowed to differ on — the AWS region and the role ARN — so any remaining output is a mistake:

mask() {
  sed -E -e 's/[a-z0-9.-]+\.rds\.amazonaws\.com/RDS_ENDPOINT/g' \
         -e 's/(us|eu|ap)-[a-z]+-[0-9]/AWS_REGION/g' \
         -e 's#arn:aws:iam::[0-9]+:role/[A-Za-z0-9_+=,.@-]+#KEK_ROLE#g' "$1"
}
diff <(mask catalyst-west-values.yaml) <(mask catalyst-east-values.yaml)

That diff should be empty.

Then, as in region 1:

make apply REGION=us-east-1 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

make output gateway_nlb_arn

diagrid region update catalyst-east \
  --ingress "<value-from-region_ingress_endpoint-output>"

This apply also writes catalyst-east.catalyst.example.com, the name that reaches this region alone. Each region creates its own; a group member needs one because the wildcard resolves to the shared entry point rather than to the region itself. Keep this region's gateway_nlb_arn as well.

Both regions now carry the same ingress domain. Issue this region's wildcard certificate in the same way, from the shared hosted zone.

If you created a shared root in step 1, verify before creating the group that the root fingerprint is the same in both clusters:

kubectl -n root-dapr-system get configmap dapr-trust-bundle \
  -o jsonpath='{.data.ca\.crt}' | openssl x509 -noout -fingerprint -sha256

7. Create the shared entry point

Both regions are now running with their own gateway load balancers, and nothing yet resolves *.catalyst.example.com. The entry point is a third Terraform state: one Global Accelerator over the two load balancers, and the wildcard record that points at it.

From catalyst-west/guides/aws — one of the two clones, and the same one every time — put this in terraform/region-group/terraform.tfvars:

region_ingress_endpoint = "catalyst.example.com"
route53_zone_id         = "Z0123456789ABCDEFGHIJ"

primary_aws_region   = "us-west-2"
primary_cluster_name = "catalyst-west"

secondary_aws_region   = "us-east-1"
secondary_cluster_name = "catalyst-east"

# Each region's gateway_nlb_arn output. With both set, this stack reads
# nothing from either member region and can still be applied while one of them
# is unreachable, which is exactly when its traffic dial is needed. If they are
# omitted, each region's load balancer is discovered by tag instead, which
# requires that region to respond.
primary_gateway_lb_arn   = "arn:aws:elasticloadbalancing:us-west-2:111122223333:loadbalancer/net/k8s-craagent-gateway/abc123"
secondary_gateway_lb_arn = "arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/net/k8s-craagent-gateway/def456"

make group-init
make group-plan
make group-apply

make group-output accelerator_ip_addresses
make group-output region_endpoints

This creates:

One accelerator with two static anycast addresses, forwarding TCP 443 to both regions and nothing else. The gateway terminates TLS itself, so the accelerator passes the connection through rather than terminating it. accelerator_ip_addresses holds the addresses to allowlist in a firewall; they survive every failover. listener_port changes the port if your gateway serves another one.
*.catalyst.example.com, aliased to the accelerator. Every project hostname and every region service name under the domain now resolves to it.

It does not create the per-region names. catalyst-west.catalyst.example.com and catalyst-east.catalyst.example.com are aliased directly to that region's load balancer, bypassing the accelerator, and each region writes its own in the apply already run. They resolve whether or not the region is receiving traffic, which is what makes it possible to query a passive region. They live in each region's state so that this stack requires nothing from a member region except its load balancer ARN.

Check that each region answers for itself:

for r in catalyst-west catalyst-east; do
  printf '%s: ' "$r"
  curl -so /dev/null -w '%{http_code}\n' "https://$r.$INGRESS_DOMAIN/diagrid/region/writable"
done

Both answer 404 at this point, and that is correct. A region publishes the health check endpoint only while it is a member of a group, and the group does not exist until the next step. Until then every target is unhealthy in both regions and the accelerator has no healthy endpoint to prefer, so the region a connection reaches is undefined. That has no impact here, because no project exists yet.

A 000 or a TLS error instead means the wildcard certificate has not been issued or the name has not propagated. Resolve that before continuing.

8. Create the region group

With both regions registered, running, and still empty:

diagrid regiongroup create my-group --regions catalyst-west,catalyst-east

9. Create a project

Create projects in the group, never before it exists:

diagrid project create my-app --region-group my-group

diagrid project get my-app --output json | jq '.status.placements'

A project in the group has a placement in each region and one set of hostnames — http-prj<id>.catalyst.example.com and its grpc- equivalent — that resolve through the accelerator to whichever region is live. Applications hold a single endpoint and are unaware that a failover has occurred.

Resources created while the group is mid-failover remain pending on the passive side and are provisioned once writes are accepted again.

10. Verify the group

Run these four checks before the group carries anything you care about.

1. The control plane reports no mismatches. This is the first point at which it can compare the two members, and it is where a mismatched KEK or an unshared PKI root is reported:

diagrid regiongroup get my-group --output json | jq '.status.messages'

An empty list is the expected result. region catalyst-east does not resolve the KEK of region catalyst-west means the two regions were configured with different key identities. Correct it now, while no project has written a secret. Afterwards the two regions hold rows that neither can fully read.

2. Exactly one member accepts writes.

diagrid regiongroup get my-group --output json | jq '.status'

The status names which member currently accepts writes. A member that reports nothing is not reporting false: its agent has stopped heartbeating, which is a different problem.

3. Each region answers the health check correctly. Re-run the loop from step 7. The active region now answers 200 and the passive region answers 503. The passive region's load balancer targets become unhealthy within about 20 seconds, and the accelerator settles on the active region.

for r in catalyst-west catalyst-east; do
  printf '%s: ' "$r"
  curl -so /dev/null -w '%{http_code}\n' "https://$r.$INGRESS_DOMAIN/diagrid/region/writable"
done

4. Replication lag is being scraped. Every passive member exports cra_region_replication_lag_seconds. That metric is your RPO, so confirm it reaches your monitoring and set an alert on it before you go live.

11. Rehearse a failover

Do this before the group serves production, not after.

A full round trip through AWS failover and failback rebuilds both regions' databases, so plan the rehearsal as maintenance rather than as an operation you can undo. Rehearse:

A planned failover from region 1 to region 2, timing each step.
A failback to region 1, so the group ends where it started.
An unplanned failover, simulated by setting the active region's traffic dial to 0 and promoting the other region without draining first.

Record the wall-clock time from "we decided to promote" to "the first write succeeded in the new region". That number is your real RTO, and it is mostly decision time rather than machine time.

Limitations

These are the limits of this Terraform, on top of the limitations of a region group itself:

It does not promote anything on its own. Nothing here reacts to a region becoming unwritable except the load balancer health check, and that only moves traffic.
It does not demote anything. RDS cannot turn a primary back into a replica, so the only demotion is destroying that region's databases and rebuilding them as replicas of the new primary.
It does not bound replication lag, though the Catalyst agent does measure it.
It does not fail over managed Kafka pub/sub or Redis rate-limit counters, which are regional. A project in a group brings its own pub/sub on a multi-region broker.
It does not size the two regions differently. Both regions take the same sizing variables: a passive region is a full region.

Next steps

🔁

Architecture​

Requirements checklist​

Before you start​

1. Decide the shared values​

The key encryption key​

Optional: one shared PKI root​

2. Create both regions in Diagrid Cloud​

3. Deploy the first region's AWS infrastructure​

4. Install Catalyst in the first region​

Optional: this region's PKI intermediate​

The values file and the Helm install​

5. Deploy the second region's AWS infrastructure​

6. Install Catalyst in the second region​

7. Create the shared entry point​

8. Create the region group​

9. Create a project​

10. Verify the group​

11. Rehearse a failover​

Limitations​

Next steps​

AWS failover and failback

Multi-region high availability

Production planning