Azure multi-region deployment

This guide builds a two-region region group on Azure and places a single global load balancer in front of both regions.

Read Multi-region high availability first. It covers what a group is, what its members must share, and why promotion is a decision you make rather than one the platform makes. This page is the Azure realization of it: PostgreSQL Flexible Server cross-region read replicas, and an Azure cross-region Load Balancer as the global load balancer.

Once the group is built, Azure failover and failback is the runbook you operate it with.

This is not the Azure installation guide's architecture

The Azure deployment guide builds a private region behind an internal load balancer. A region built that way cannot join a region group, because an Azure cross-region load balancer accepts only public frontends. This page builds public regions from its own Terraform stack, in guides/azure/terraform.

Architecture

Every client reaches the group at the front door's single static address, so a failover moves traffic without any client re-resolving a name or waiting for a DNS TTL.

Each region's load balancer probes the region's gateway, which answers 200 while its database accepts writes and 503 while it is a replica. The front door sends traffic to whichever region reports healthy, so promoting the replica is what moves traffic.

Ensuring that exactly one region holds the primary at any time is your responsibility. Nothing in the design enforces it.

Both regions run a full Catalyst data plane. Size both identically, and budget for two.

Requirements checklist

The region group requirements list six settings the members must share. The control plane enforces none of them. This guide satisfies them as follows:

Requirement	How this guide provides it	Where you set it
One replicated database	A PostgreSQL Flexible Server in the first region, and a cross-region read replica of it in the second	postgresql_replicate_source_server_id (Terraform)
The same database name and admin user	Inherited: a Flexible Server replica takes both from the server it follows	—
The PostgreSQL secrets provider	Secrets live in the replicated database instead of in one cluster's Kubernetes	global.secrets.provider (Helm)
The same key encryption key	The same 64-hex key configured in both regions	global.secrets.postgresql.primary_encryption_key (Helm)
The PostgreSQL scheduler backend	Jobs and actor reminders live in the replicated database, so they move with it	scheduler_postgresql_instances = [] (Terraform), agent.config.internal_dapr.scheduler.postgresql.use_global (Helm)
One Dapr PKI root — optional	A root you create once for the group, with an intermediate per region	agent.config.internal_dapr.pki (Helm)

Before you start

• Two Azure regions in one subscription, and a wildcard domain you control. The two regions can be any pair — the restriction is on the front door, in step 8.
• vCPU quota for two AKS clusters and two Flexible Servers at your chosen tier, in both regions. If an apply fails on the subscription rather than on your configuration, see Troubleshooting.
• Separate Terraform state per region. You apply the same stack twice, from two working directories. A single configuration holding both regions cannot produce a plan while one region's API is unreachable, and that is exactly when you need to promote.
• Empty regions. A region that already holds a project cannot join a group. Create both regions, group them, then create your projects.
• A new project. You cannot move an existing single-region project into a group. That means new hostnames and API keys, and a redeploy of the applications that call it.
• A decision about workflow history signing, made in step 1. It cannot be added after the regions join.
• Tools: the Azure CLI, Terraform, Helm, kubectl, kubelogin (install it with az aks install-cli — brew install kubelogin installs a different tool of the same name), jq, and the diagrid CLI v1.87.0 or later.

Set up one working directory per region:

git clone https://github.com/diagridio/charts.git catalyst-west
git clone https://github.com/diagridio/charts.git catalyst-east

1. Decide the shared values

Decide these once, before either region exists.

# The wildcard domain both regions serve.
export INGRESS_DOMAIN="catalyst.example.com"

# The database admin password. Set on the first region only — the replica
# inherits it — but both regions' Helm values need it. Keep it out of
# terraform.tfvars.
export PGPASSWORD="<a strong password>"

The key encryption key

Catalyst encrypts every secret it stores with a key encryption key. The encrypted rows replicate between the regions, so both regions must hold the same key, or the passive region cannot decrypt them after a promotion. Generate it once:

openssl rand -hex 32

Both regions' Helm values set this as primary_encryption_key, byte for byte.

The key is held in both clusters

Azure has no managed key encryption key provider in Catalyst yet, so the key sits in a Kubernetes secret in both clusters. Anyone who can read secrets in either cluster's cra-agent namespace can decrypt every Catalyst secret in the group. Store it in a secret manager, distribute it out of band, and keep it out of values files — see the tip in step 4.

Optional: one shared PKI root

Only if any project in the group will use workflow history signing. It must be in place before the regions first join. Follow the AWS guide's PKI step — nothing about it is cloud-specific.

2. Create both regions in Diagrid Cloud

export JOIN_TOKEN_WEST=$(diagrid region create catalyst-west \
  --ingress "https://*.$INGRESS_DOMAIN:443" -o json | jq -r .joinToken)

export JOIN_TOKEN_EAST=$(diagrid region create catalyst-east \
  --ingress "https://*.$INGRESS_DOMAIN:443" -o json | jq -r .joinToken)

Keep both join tokens. Both regions get the same ingress, because they serve the same domain.

3. Deploy the first region's Azure infrastructure

From catalyst-west/guides/azure, with this terraform/terraform.tfvars:

subscription_id = "00000000-0000-0000-0000-000000000000"
tenant_id       = "11111111-1111-1111-1111-111111111111"

cluster_name = "catalyst-west"

postgresql_password = "<the password from step 1>"

# Who may use the cluster. This is the only access the stack grants, so an
# empty list builds a cluster nobody can use. Your own object id:
#   az ad signed-in-user show --query id -o tsv
aks_admin_principal_ids = ["<your Entra ID object id>"]

# The shared entry point owns the wildcard record, not this region.
region_group_member = true

# Keep the Dapr scheduler's state in the shared database.
scheduler_postgresql_instances = []

To size the region, copy the values from one of the tier files into this terraform.tfvars. Both regions must be the same tier: a replica cannot be smaller than the server it follows.

make init
make plan  LOCATION=westus2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply LOCATION=westus2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

Delegate the domain to the nameservers in make output dns_zone_name_servers at your registrar. Certificates cannot be issued until the delegation has taken effect.

Connect to the cluster

az aks get-credentials --resource-group catalyst-west-rg --name catalyst-west
kubelogin convert-kubeconfig -l azurecli

Everything from here to the end of step 4 runs against this cluster.

Issue the wildcard certificate

The gateway serves TLS for *.catalyst.example.com from a secret named cert-wildcard, and stays unready until that secret exists. cert-manager issues it with a DNS-01 challenge, using an Azure identity the Terraform created for it:

export CERT_MANAGER_CLIENT_ID=$(make output cert_manager_identity_client_id)
export DNS_ZONE_RG=$(make output cert_manager_dns_zone_resource_group_name)

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true \
  --set "serviceAccount.annotations.azure\.workload\.identity/client-id=$CERT_MANAGER_CLIENT_ID" \
  --set-string "podLabels.azure\.workload\.identity/use=true"

Create the issuer:

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <your email address>
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - dns01:
          azureDNS:
            hostedZoneName: $INGRESS_DOMAIN
            resourceGroupName: $DNS_ZONE_RG
            subscriptionID: <your subscription id>
            environment: AzurePublicCloud
            managedIdentity:
              clientID: $CERT_MANAGER_CLIENT_ID
EOF

Then request the certificate. Create the cra-agent namespace first — a Certificate in a namespace that does not exist never issues:

kubectl create namespace cra-agent

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cert-wildcard
  namespace: cra-agent
spec:
  secretName: cert-wildcard
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  dnsNames:
    - "*.$INGRESS_DOMAIN"
EOF

kubectl wait --for=condition=Ready certificate/cert-wildcard -n cra-agent --timeout=5m

The first issue usually takes a few minutes. If it stays False, kubectl describe certificaterequest -n cra-agent names the reason; a challenge stuck on presenting is nearly always the delegation not having taken effect yet.

4. Install Catalyst in the first region

Optional: this region's PKI intermediate

If you decided on workflow history signing in step 1, install this region's intermediate now, exactly as the AWS guide's step describes.

Let the database user stream changes

The Dapr scheduler needs the database administrator to have the REPLICATION attribute, which Flexible Server does not grant by default and Terraform cannot set. The server is reachable only from inside the virtual network, so grant it from a pod in the cluster:

kubectl run grant-replication --rm -i --restart=Never -n default \
  --image=postgres:17-alpine --env="PGPASSWORD=$PGPASSWORD" --command -- \
  psql "host=$(make output postgresql_endpoint) user=postgres dbname=catalyst sslmode=require" \
  -c "ALTER ROLE postgres WITH REPLICATION;"

Do this once, on this region only. The attribute replicates to the second region with the rest of the database.

The values file and the Helm install

export PG_ENDPOINT=$(make output postgresql_endpoint)
export RESOURCE_GROUP=$(make output resource_group_name)
export GATEWAY_PIP_NAME=$(make output gateway_public_ip_name)
export KEK_KEY="<the 64-hex key from step 1>"
export JOIN_TOKEN="$JOIN_TOKEN_WEST"

cat > catalyst-values.yaml << EOF
global:
  # SHARED: secrets live in the replicated database.
  secrets:
    provider: postgresql
    postgresql:
      connection_string: "postgres://postgres:$PGPASSWORD@$PG_ENDPOINT:5432/catalyst"
      # SHARED: the same key in both regions, byte for byte.
      kek_provider: local
      primary_encryption_key: "$KEK_KEY"
      primary_key_version: 1

agent:
  config:
    project:
      # SHARED: project state in the replicated database. The host is this
      # region's own server.
      default_managed_state_store_type: postgresql-shared-external
      external_postgresql:
        enabled: true
        auth_type: connectionString
        namespace: postgresql
        connection_string_host: $PG_ENDPOINT
        connection_string_port: 5432
        connection_string_username: postgres
        connection_string_password: "$PGPASSWORD"
    internal_dapr:
      scheduler:
        # SHARED: jobs and actor reminders in the replicated database, so they
        # survive a failover.
        backend_type: postgresql
        postgresql:
          use_global: true

gateway:
  tls:
    enabled: true
    secretName: "cert-wildcard"
  envoy:
    service:
      type: LoadBalancer
      httpsPort: 443
      httpsTargetPort: 8443
      annotations:
        # Claim the public address Terraform created for this region.
        service.beta.kubernetes.io/azure-load-balancer-resource-group: "$RESOURCE_GROUP"
        service.beta.kubernetes.io/azure-pip-name: "$GATEWAY_PIP_NAME"
        # The failover signal: the region is healthy while its database
        # accepts writes. One probe per port the Service publishes.
        service.beta.kubernetes.io/port_443_health-probe_protocol: "https"
        service.beta.kubernetes.io/port_443_health-probe_request-path: "/diagrid/region/writable"
        service.beta.kubernetes.io/port_8080_health-probe_protocol: "http"
        service.beta.kubernetes.io/port_8080_health-probe_request-path: "/diagrid/region/writable"
        service.beta.kubernetes.io/azure-load-balancer-health-probe-interval: "5"
        service.beta.kubernetes.io/azure-load-balancer-health-probe-num-of-probe: "2"
EOF

helm install catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent \
     --create-namespace \
     -f catalyst-values.yaml \
     --set join_token="${JOIN_TOKEN}"

Do not set externalTrafficPolicy: Local on the gateway Service. It makes AKS ignore the probe path, so both regions report healthy and traffic reaches the replica.

Keep the database password and the key out of the values file

The inline form above shows the whole shared configuration in one place. In production, put the secrets provider's configuration in a Kubernetes Secret and set global.secrets.postgresql.existingSecret, as the Helm chart reference describes. On Azure primary_encryption_key is the key itself, not a reference to one.

This region is unreachable until the group exists

The gateway answers /diagrid/region/writable for group members only. Until step 6 creates the group, the probe fails and the load balancer drops every connection to the region's address. diagrid region list still shows the region online, because the agent connects out.

5. Deploy the second region's Azure infrastructure

From catalyst-east/guides/azure, with this terraform/terraform.tfvars. Copy the first region's tier values in as well.

subscription_id = "00000000-0000-0000-0000-000000000000"
tenant_id       = "11111111-1111-1111-1111-111111111111"

cluster_name = "catalyst-east"

# Unused while this region is a replica, but required by the stack.
postgresql_password = "<the password from step 1>"

aks_admin_principal_ids        = ["<your Entra ID object id>"]
region_group_member            = true
scheduler_postgresql_instances = []

# Join region 1's DNS zone instead of creating a second one.
dns_zone_resource_group_name = "catalyst-west-rg"

# Build this region's database as a read replica of region 1's
# (make output postgresql_server_id in catalyst-west). Never clear this.
postgresql_replicate_source_server_id = "<region 1's postgresql_server_id>"

# A read replica cannot have high availability. Turn it on after a promotion.
postgresql_high_availability = false

# The two networks are peered for replication, so they must not overlap.
vnet_cidr            = "10.1.0.0/16"
aks_subnet_cidr      = "10.1.1.0/24"
database_subnet_cidr = "10.1.2.0/24"
bastion_subnet_cidr  = "10.1.3.0/24"

# Peer with region 1 (make output vnet_id in catalyst-west). Leave it set for
# the group's lifetime.
region_group_peer_vnet_id = "<region 1's vnet_id>"

make init
make plan  LOCATION=eastus2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply LOCATION=eastus2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN

Never clear postgresql_replicate_source_server_id

Clearing it destroys this region's database rather than promoting it. Promote with failover.sh promote, which leaves it set — a promoted server keeps its source server id for life.

6. Create the region group

Create the group now, before the second region's Catalyst install:

diagrid region group create my-group --regions catalyst-west,catalyst-east

The second region's database is read-only from the start, and its agent only knows to treat it that way if the region is already a group member when it first connects.

7. Install Catalyst in the second region

az aks get-credentials --resource-group catalyst-east-rg --name catalyst-east
kubelogin convert-kubeconfig -l azurecli

Run Issue the wildcard certificate against this cluster, from catalyst-east/guides/azure. The two make output values it reads are this region's own; cert_manager_dns_zone_resource_group_name already points at region 1's zone. There is no second delegation to do, and no replication grant — the replica already has it.

Then create the values file and install exactly as in step 4, with JOIN_TOKEN="$JOIN_TOKEN_EAST". The three make output values are this region's own; everything marked SHARED stays identical, including primary_encryption_key.

8. Create the shared entry point

The entry point is a second Terraform state: one cross-region load balancer over the two regions' gateways, and the wildcard record that points at it.

From catalyst-west/guides/azure — and always from the same clone — put this in terraform/region-group/terraform.tfvars:

subscription_id = "00000000-0000-0000-0000-000000000000"
tenant_id       = "11111111-1111-1111-1111-111111111111"

region_ingress_endpoint      = "catalyst.example.com"
dns_zone_resource_group_name = "catalyst-west-rg"

# One of Azure's cross-region load balancer home regions (see Limitations).
# It need not be either member's region, and takes no part in routing.
location = "westus"

# From make output in each region's working directory.
primary_cluster_name         = "catalyst-west"
primary_node_resource_group  = "<region 1's aks_node_resource_group>"
primary_gateway_public_ip_id = "<region 1's gateway_public_ip_id>"

secondary_cluster_name         = "catalyst-east"
secondary_node_resource_group  = "<region 2's aks_node_resource_group>"
secondary_gateway_public_ip_id = "<region 2's gateway_public_ip_id>"

make group-init
make group-plan
make group-apply

make group-output front_door_ip_address
make group-output gateway_frontend_ip_configurations

This creates the cross-region load balancer, forwarding TCP 443 to both regions, and *.catalyst.example.com pointing at its address. front_door_ip_address is the address to allowlist in a firewall; it survives every failover.

Record the frontend ids now

Put the two ids from gateway_frontend_ip_configurations into terraform.tfvars as primary_gateway_frontend_ip_configuration_id and secondary_gateway_frontend_ip_configuration_id. With both set, this stack can still be applied while one region is down — which is exactly when you need to drain it.

Admit the front door's address in both regions

Traffic relayed by the front door arrives addressed to the front door's address, and AKS accepts it only if the gateway Service names that address. Add it to both regions' values files:

gateway:
  envoy:
    service:
      annotations:
        # From make group-output front_door_ip_address.
        service.beta.kubernetes.io/azure-additional-public-ips: "<front_door_ip_address>"

and upgrade each release against its own cluster:

helm upgrade catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent -f catalyst-values.yaml --set join_token="${JOIN_TOKEN}"

The address never changes, so this is done once.

Check the entry point

for r in catalyst-west catalyst-east; do
  printf '%s: ' "$r"
  curl -so /dev/null -m 10 -w '%{http_code}\n' "https://$r.$INGRESS_DOMAIN/diagrid/region/writable"
done
printf 'front door: '
curl -so /dev/null -m 10 -w '%{http_code}\n' "https://any-name.$INGRESS_DOMAIN/diagrid/region/writable"

Region 1 and the front door answer 200. Region 2 gives no answer (000): a passive region's load balancer drops every connection, so it cannot be reached at its own name from outside. Anything else, see Troubleshooting.

9. Create a project

Create projects in the group, never before it exists:

diagrid project create my-app --region-group my-group

diagrid project get my-app --output json | jq '.status.instances'

Each entry names the region it is placed in and its readiness. The project has one set of hostnames — http-prj<id>.catalyst.example.com and its grpc- equivalent — that resolve through the front door to whichever region is live. Applications hold a single endpoint and are unaware that a failover has occurred.

10. Verify the group

Run these checks before the group carries anything you care about.

1. The control plane reports one writer and no mismatches.

diagrid region group get my-group --output json | jq '.status'

Exactly one member accepts writes. A member that reports nothing has stopped heartbeating.

In messages, if you skipped the shared PKI root, the one expected entry is members do not share a PKI root (…). With a shared root the list is empty. region catalyst-east does not resolve the KEK of region catalyst-west means the two regions were given different primary_encryption_key values — correct it now, before any project writes a secret.

2. Each region agrees with the control plane. Re-run the entry point check: the writer answers 200, the other gives no answer, and the front door answers 200.

3. Replication lag is being scraped. Every passive member exports cra_region_replication_lag_seconds. That metric is your RPO, so confirm it reaches your monitoring and set an alert on it before you go live.

11. Rehearse a failover

Do this before the group serves production, not after.

A full round trip through Azure failover and failback rebuilds both regions' databases, so plan the rehearsal as maintenance. Rehearse:

• A planned failover from region 1 to region 2, timing each step.
• A failback to region 1, so the group ends where it started.
• An unplanned failover, simulated by taking the active region's gateway away — kubectl scale deploy catalyst-gateway -n cra-agent --replicas=0 — and running Recover with no drain first. Scaling it back afterwards is the lost region returning as a second writer, which the runbook's rebuild then resolves.

Record the wall-clock time from "we decided to promote" to "the first write succeeded in the new region". That number is your real RTO, and it is mostly decision time rather than machine time.

Tear the group down

Follow this order. Deleting things in a different order fails partway through.

1. Delete the projects, then the group. diagrid project delete, then diagrid region group delete my-group. The member regions are not deleted with the group.
2. Destroy the shared entry point: make group-destroy, from the working directory that holds its state.
3. helm uninstall in each region, so the gateway releases the region's public address while the cluster still exists.
4. End the replication. Whichever region holds the replica, promote it with failover.sh promote.
5. Destroy region 2, then region 1 — always in that order, even after a failover. Region 2 owns the network peering and records inside region 1's DNS zones, which region 1 cannot delete until they are gone.

Troubleshooting

Symptom	Cause and fix
az vm list-usage returns nothing, or an apply fails with MissingSubscriptionRegistration	The subscription is new and has no resource providers registered. Run az provider register -n <namespace> for Microsoft.Compute, Microsoft.ContainerService, Microsoft.DBforPostgreSQL, Microsoft.Network, Microsoft.OperationalInsights and Microsoft.ManagedIdentity, and wait until az provider show -n <namespace> --query registrationState reports Registered.
The apply fails on vCPU quota	Quota is counted per VM family as well as in total, per region. az vm list-usage -l <region> -o table shows both. If your node size's family has no row at all, it is not offered to your subscription and you need a different size.
The cluster create fails on zone placement	The node size is not available in the region's zones for your subscription. az vm list-skus -l <region> --size <size> --zone -o table returning nothing confirms it: set availability_zones = [], or to the zones it lists, in both regions.
HA is disabled for region <region>	The region does not offer zone-redundant PostgreSQL high availability to your subscription. Set postgresql_high_availability = false for that region.
K8sVersionNotSupported	The stack's cluster_version default has moved to Long-Term Support only. Pick a version shown as KubernetesOfficial in az aks get-versions -l <region> -o table in both regions, and set it in both.
kubectl fails with executable kubelogin not found	Install kubelogin with az aks install-cli, not Homebrew, and run kubelogin convert-kubeconfig -l azurecli.
kubectl fails with User does not have access to the resource in Azure	Your object id is not in aks_admin_principal_ids. Add it and re-apply. If it was already there, kubelogin is holding an old token: rm -rf ~/.kube/cache/kubelogin.
helm install cert-manager fails with cannot unmarshal bool	The pod label was passed with --set. It must be --set-string.
The Dapr scheduler never becomes ready and the agent restarts every five minutes, with permission denied to start WAL sender	The REPLICATION grant was not made on region 1's database.
The second region's agent exits with cannot execute CREATE TABLE in a read-only transaction	The region joined before the group existed. It recovers on its own on the next restart once the group exists.
Region 1 gives no answer (000) at its own name	It is not a group member yet (diagrid region group get my-group), the certificate has not been issued, or the name has not propagated. A TLS error instead of 000 is the certificate.
Region 1 answers, the front door gives no answer	The gateway Service is missing the front door's address.

For problems after the group is running, see the failover runbook's troubleshooting.

Limitations

These are the limits of this Terraform, on top of the limitations of a region group itself:

• The key encryption key is held in both clusters, not outside them. Azure has no managed key encryption key provider in Catalyst yet. See the key encryption key.
• Draining is all or nothing. A cross-region load balancer has no traffic percentages, so draining a region removes it from the backend pool.
• The front door must live in one of ten home regions — Central US, East Asia, East US 2, North Europe, Southeast Asia, UK South, US Gov Virginia, West Europe, West US, or China North 2. The two member regions can be any pair.
• It does not bound replication lag, though the Catalyst agent measures it.
• Both regions must be in one subscription. The entry point stack uses a single provider for both members.

Next steps

🔁

Azure failover and failback

The runbook: promote a region, move traffic, and rebuild the old one as a replica.

🌍

Multi-region high availability

What a region group is, what its members share, and how a failover works on any cloud.

📐

Production planning

Size, secure, and operate a self-managed region before it takes production traffic.