# Azure multi-region deployment

This guide builds a two-region [region group](https://docs.diagrid.io/operate/platform-operations/multi-region) on Azure and places a single global load balancer in front of both regions.

Read [Multi-region high availability](https://docs.diagrid.io/operate/platform-operations/multi-region) first. It covers what a group is, what its members must share, and why promotion is a decision you make rather than one the platform makes. This page is the Azure realization of it: PostgreSQL Flexible Server cross-region read replicas, and an Azure cross-region Load Balancer as the global load balancer.

Once the group is built, [Azure failover and failback](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-failover) is the runbook you operate it with.

:::info This is not the Azure installation guide's architecture

The [Azure deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-installation-guide) guide builds a private region behind an internal load balancer. **A region built that way cannot join a region group**, because an Azure cross-region load balancer accepts only public frontends. This page builds public regions from its own Terraform stack, in [`guides/azure/terraform`](https://github.com/diagridio/charts/tree/main/guides/azure/terraform).

:::

## Architecture

```mermaid
---
title: Two Azure regions in one Catalyst region group
---
flowchart TD
  DNS("Azure DNS<br/>*.catalyst.example.com")
  GLB("Cross-region Load Balancer<br/>one static anycast address")

  DNS-->GLB

  subgraph Active["Azure region 1 — active"]
    LB1("Cluster load balancer<br/>health probe: /diagrid/region/writable")
    AKS1("AKS — Catalyst data plane")
    PG1[("PostgreSQL Flexible Server<br/>primary")]
  end

  subgraph Passive["Azure region 2 — passive"]
    LB2("Cluster load balancer<br/>health probe: /diagrid/region/writable")
    AKS2("AKS — Catalyst data plane")
    PG2[("PostgreSQL Flexible Server<br/>read replica")]
  end

  GLB=="probe healthy (200)"==>LB1
  GLB-."probe unhealthy (503)".->LB2
  LB1-->AKS1
  LB2-->AKS2
  AKS1-->PG1
  AKS2-->PG2
  PG1=="cross-region replication"==>PG2
```

Every client reaches the group at the front door's single static address, so a failover moves traffic without any client re-resolving a name or waiting for a DNS TTL.

Each region's load balancer probes the region's gateway, which answers `200` while its database accepts writes and `503` while it is a replica. The front door sends traffic to whichever region reports healthy, so promoting the replica is what moves traffic.

Ensuring that exactly one region holds the primary at any time is your responsibility. Nothing in the design enforces it.

Both regions run a full Catalyst data plane. Size both identically, and budget for two.

## Requirements checklist

[The region group requirements](https://docs.diagrid.io/operate/platform-operations/multi-region#requirements) list six settings the members must share. The control plane enforces none of them. This guide satisfies them as follows:

| Requirement | How this guide provides it | Where you set it |
|---|---|---|
| One replicated database | A PostgreSQL Flexible Server in the first region, and a cross-region read replica of it in the second | `postgresql_replicate_source_server_id` (Terraform) |
| The same database name and admin user | Inherited: a Flexible Server replica takes both from the server it follows | — |
| The PostgreSQL secrets provider | Secrets live in the replicated database instead of in one cluster's Kubernetes | `global.secrets.provider` (Helm) |
| The same key encryption key | The same 64-hex key configured in both regions | `global.secrets.postgresql.primary_encryption_key` (Helm) |
| The PostgreSQL scheduler backend | Jobs and actor reminders live in the replicated database, so they move with it | `scheduler_postgresql_instances = []` (Terraform), `agent.config.internal_dapr.scheduler.postgresql.use_global` (Helm) |
| One Dapr PKI root — *optional* | A root you create once for the group, with an intermediate per region | `agent.config.internal_dapr.pki` (Helm) |

## Before you start

Set up one working directory per region:

```bash
git clone https://github.com/diagridio/charts.git catalyst-west
git clone https://github.com/diagridio/charts.git catalyst-east
```

## 1. Decide the shared values

Decide these once, before either region exists.

```bash
# The wildcard domain both regions serve.
export INGRESS_DOMAIN="catalyst.example.com"

# The database admin password. Set on the first region only — the replica
# inherits it — but both regions' Helm values need it. Keep it out of
# terraform.tfvars.
export PGPASSWORD="<a strong password>"
```

### The key encryption key

Catalyst encrypts every secret it stores with a key encryption key. The encrypted rows replicate between the regions, so **both regions must hold the same key**, or the passive region cannot decrypt them after a promotion. Generate it once:

```bash
openssl rand -hex 32
```

Both regions' Helm values set this as `primary_encryption_key`, byte for byte.

:::caution The key is held in both clusters

Azure has no managed key encryption key provider in Catalyst yet, so the key sits in a Kubernetes secret in **both** clusters. Anyone who can read secrets in either cluster's `cra-agent` namespace can decrypt every Catalyst secret in the group. Store it in a secret manager, distribute it out of band, and keep it out of values files — see the [tip in step 4](#the-values-file-and-the-helm-install).

:::

### Optional: one shared PKI root

Only if any project in the group will use workflow history signing. It must be in place **before** the regions first join. Follow [the AWS guide's PKI step](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-deployment#optional-one-shared-pki-root) — nothing about it is cloud-specific.

## 2. Create both regions in Diagrid Cloud

```bash
export JOIN_TOKEN_WEST=$(diagrid region create catalyst-west \
  --ingress "https://*.$INGRESS_DOMAIN:443" -o json | jq -r .joinToken)

export JOIN_TOKEN_EAST=$(diagrid region create catalyst-east \
  --ingress "https://*.$INGRESS_DOMAIN:443" -o json | jq -r .joinToken)
```

Keep both join tokens. Both regions get the same ingress, because they serve the same domain.

## 3. Deploy the first region's Azure infrastructure

From `catalyst-west/guides/azure`, with this `terraform/terraform.tfvars`:

```hcl
subscription_id = "00000000-0000-0000-0000-000000000000"
tenant_id       = "11111111-1111-1111-1111-111111111111"

cluster_name = "catalyst-west"

postgresql_password = "<the password from step 1>"

# Who may use the cluster. This is the only access the stack grants, so an
# empty list builds a cluster nobody can use. Your own object id:
#   az ad signed-in-user show --query id -o tsv
aks_admin_principal_ids = ["<your Entra ID object id>"]

# The shared entry point owns the wildcard record, not this region.
region_group_member = true

# Keep the Dapr scheduler's state in the shared database.
scheduler_postgresql_instances = []
```

To size the region, copy the values from one of the [tier files](https://github.com/diagridio/charts/tree/main/guides/azure/terraform/tiers) into this `terraform.tfvars`. **Both regions must be the same tier**: a replica cannot be smaller than the server it follows.

```bash
make init
make plan  LOCATION=westus2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply LOCATION=westus2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
```

Delegate the domain to the nameservers in `make output dns_zone_name_servers` at your registrar. Certificates cannot be issued until the delegation has taken effect.

### Connect to the cluster

```bash
az aks get-credentials --resource-group catalyst-west-rg --name catalyst-west
kubelogin convert-kubeconfig -l azurecli
```

Everything from here to the end of step 4 runs against this cluster.

### Issue the wildcard certificate

The gateway serves TLS for `*.catalyst.example.com` from a secret named `cert-wildcard`, and stays unready until that secret exists. cert-manager issues it with a DNS-01 challenge, using an Azure identity the Terraform created for it:

```bash
export CERT_MANAGER_CLIENT_ID=$(make output cert_manager_identity_client_id)
export DNS_ZONE_RG=$(make output cert_manager_dns_zone_resource_group_name)

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true \
  --set "serviceAccount.annotations.azure\.workload\.identity/client-id=$CERT_MANAGER_CLIENT_ID" \
  --set-string "podLabels.azure\.workload\.identity/use=true"
```

Create the issuer:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <your email address>
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - dns01:
          azureDNS:
            hostedZoneName: $INGRESS_DOMAIN
            resourceGroupName: $DNS_ZONE_RG
            subscriptionID: <your subscription id>
            environment: AzurePublicCloud
            managedIdentity:
              clientID: $CERT_MANAGER_CLIENT_ID
EOF
```

Then request the certificate. Create the `cra-agent` namespace first — a `Certificate` in a namespace that does not exist never issues:

```bash
kubectl create namespace cra-agent

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cert-wildcard
  namespace: cra-agent
spec:
  secretName: cert-wildcard
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  dnsNames:
    - "*.$INGRESS_DOMAIN"
EOF

kubectl wait --for=condition=Ready certificate/cert-wildcard -n cra-agent --timeout=5m
```

The first issue usually takes a few minutes. If it stays `False`, `kubectl describe certificaterequest -n cra-agent` names the reason; a challenge stuck on `presenting` is nearly always the delegation not having taken effect yet.

## 4. Install Catalyst in the first region

### Optional: this region's PKI intermediate

If you decided on workflow history signing in step 1, install this region's intermediate now, exactly as the [AWS guide's step](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-deployment#optional-this-regions-pki-intermediate) describes.

### Let the database user stream changes

The Dapr scheduler needs the database administrator to have the `REPLICATION` attribute, which Flexible Server does not grant by default and Terraform cannot set. The server is reachable only from inside the virtual network, so grant it from a pod in the cluster:

```bash
kubectl run grant-replication --rm -i --restart=Never -n default \
  --image=postgres:17-alpine --env="PGPASSWORD=$PGPASSWORD" --command -- \
  psql "host=$(make output postgresql_endpoint) user=postgres dbname=catalyst sslmode=require" \
  -c "ALTER ROLE postgres WITH REPLICATION;"
```

Do this once, on this region only. The attribute replicates to the second region with the rest of the database.

### The values file and the Helm install

```bash
export PG_ENDPOINT=$(make output postgresql_endpoint)
export RESOURCE_GROUP=$(make output resource_group_name)
export GATEWAY_PIP_NAME=$(make output gateway_public_ip_name)
export KEK_KEY="<the 64-hex key from step 1>"
export JOIN_TOKEN="$JOIN_TOKEN_WEST"

cat > catalyst-values.yaml << EOF
global:
  # SHARED: secrets live in the replicated database.
  secrets:
    provider: postgresql
    postgresql:
      connection_string: "postgres://postgres:$PGPASSWORD@$PG_ENDPOINT:5432/catalyst"
      # SHARED: the same key in both regions, byte for byte.
      kek_provider: local
      primary_encryption_key: "$KEK_KEY"
      primary_key_version: 1

agent:
  config:
    project:
      # SHARED: project state in the replicated database. The host is this
      # region's own server.
      default_managed_state_store_type: postgresql-shared-external
      external_postgresql:
        enabled: true
        auth_type: connectionString
        namespace: postgresql
        connection_string_host: $PG_ENDPOINT
        connection_string_port: 5432
        connection_string_username: postgres
        connection_string_password: "$PGPASSWORD"
    internal_dapr:
      scheduler:
        # SHARED: jobs and actor reminders in the replicated database, so they
        # survive a failover.
        backend_type: postgresql
        postgresql:
          use_global: true

gateway:
  tls:
    enabled: true
    secretName: "cert-wildcard"
  envoy:
    service:
      type: LoadBalancer
      httpsPort: 443
      httpsTargetPort: 8443
      annotations:
        # Claim the public address Terraform created for this region.
        service.beta.kubernetes.io/azure-load-balancer-resource-group: "$RESOURCE_GROUP"
        service.beta.kubernetes.io/azure-pip-name: "$GATEWAY_PIP_NAME"
        # The failover signal: the region is healthy while its database
        # accepts writes. One probe per port the Service publishes.
        service.beta.kubernetes.io/port_443_health-probe_protocol: "https"
        service.beta.kubernetes.io/port_443_health-probe_request-path: "/diagrid/region/writable"
        service.beta.kubernetes.io/port_8080_health-probe_protocol: "http"
        service.beta.kubernetes.io/port_8080_health-probe_request-path: "/diagrid/region/writable"
        service.beta.kubernetes.io/azure-load-balancer-health-probe-interval: "5"
        service.beta.kubernetes.io/azure-load-balancer-health-probe-num-of-probe: "2"
EOF

helm install catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent \
     --create-namespace \
     -f catalyst-values.yaml \
     --set join_token="${JOIN_TOKEN}"
```

Do not set `externalTrafficPolicy: Local` on the gateway Service. It makes AKS ignore the probe path, so both regions report healthy and traffic reaches the replica.

:::tip Keep the database password and the key out of the values file

The inline form above shows the whole shared configuration in one place. In production, put the secrets provider's configuration in a Kubernetes Secret and set `global.secrets.postgresql.existingSecret`, as the [Helm chart reference](./reference.md#postgresql-secrets-provider) describes. On Azure `primary_encryption_key` is the key itself, not a reference to one.

:::

:::caution This region is unreachable until the group exists

The gateway answers `/diagrid/region/writable` for group members only. Until [step 6](#6-create-the-region-group) creates the group, the probe fails and the load balancer drops every connection to the region's address. `diagrid region list` still shows the region `online`, because the agent connects out.

:::

## 5. Deploy the second region's Azure infrastructure

From `catalyst-east/guides/azure`, with this `terraform/terraform.tfvars`. Copy the first region's tier values in as well.

```hcl
subscription_id = "00000000-0000-0000-0000-000000000000"
tenant_id       = "11111111-1111-1111-1111-111111111111"

cluster_name = "catalyst-east"

# Unused while this region is a replica, but required by the stack.
postgresql_password = "<the password from step 1>"

aks_admin_principal_ids        = ["<your Entra ID object id>"]
region_group_member            = true
scheduler_postgresql_instances = []

# Join region 1's DNS zone instead of creating a second one.
dns_zone_resource_group_name = "catalyst-west-rg"

# Build this region's database as a read replica of region 1's
# (make output postgresql_server_id in catalyst-west). Never clear this.
postgresql_replicate_source_server_id = "<region 1's postgresql_server_id>"

# A read replica cannot have high availability. Turn it on after a promotion.
postgresql_high_availability = false

# The two networks are peered for replication, so they must not overlap.
vnet_cidr            = "10.1.0.0/16"
aks_subnet_cidr      = "10.1.1.0/24"
database_subnet_cidr = "10.1.2.0/24"
bastion_subnet_cidr  = "10.1.3.0/24"

# Peer with region 1 (make output vnet_id in catalyst-west). Leave it set for
# the group's lifetime.
region_group_peer_vnet_id = "<region 1's vnet_id>"
```

```bash
make init
make plan  LOCATION=eastus2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
make apply LOCATION=eastus2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN
```

:::danger Never clear `postgresql_replicate_source_server_id`

Clearing it destroys this region's database rather than promoting it. Promote with [`failover.sh promote`](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-failover#3-promote-the-replica), which leaves it set — a promoted server keeps its source server id for life.

:::

## 6. Create the region group

Create the group now, before the second region's Catalyst install:

```bash
diagrid region group create my-group --regions catalyst-west,catalyst-east
```

The second region's database is read-only from the start, and its agent only knows to treat it that way if the region is already a group member when it first connects.

## 7. Install Catalyst in the second region

```bash
az aks get-credentials --resource-group catalyst-east-rg --name catalyst-east
kubelogin convert-kubeconfig -l azurecli
```

Run [Issue the wildcard certificate](#issue-the-wildcard-certificate) against this cluster, from `catalyst-east/guides/azure`. The two `make output` values it reads are this region's own; `cert_manager_dns_zone_resource_group_name` already points at region 1's zone. There is no second delegation to do, and no replication grant — the replica already has it.

Then create the values file and install exactly as in [step 4](#the-values-file-and-the-helm-install), with `JOIN_TOKEN="$JOIN_TOKEN_EAST"`. The three `make output` values are this region's own; everything marked `SHARED` stays identical, including `primary_encryption_key`.

## 8. Create the shared entry point

The entry point is a second Terraform state: one cross-region load balancer over the two regions' gateways, and the wildcard record that points at it.

From `catalyst-west/guides/azure` — and always from the same clone — put this in `terraform/region-group/terraform.tfvars`:

```hcl
subscription_id = "00000000-0000-0000-0000-000000000000"
tenant_id       = "11111111-1111-1111-1111-111111111111"

region_ingress_endpoint      = "catalyst.example.com"
dns_zone_resource_group_name = "catalyst-west-rg"

# One of Azure's cross-region load balancer home regions (see Limitations).
# It need not be either member's region, and takes no part in routing.
location = "westus"

# From make output in each region's working directory.
primary_cluster_name         = "catalyst-west"
primary_node_resource_group  = "<region 1's aks_node_resource_group>"
primary_gateway_public_ip_id = "<region 1's gateway_public_ip_id>"

secondary_cluster_name         = "catalyst-east"
secondary_node_resource_group  = "<region 2's aks_node_resource_group>"
secondary_gateway_public_ip_id = "<region 2's gateway_public_ip_id>"
```

```bash
make group-init
make group-plan
make group-apply

make group-output front_door_ip_address
make group-output gateway_frontend_ip_configurations
```

This creates the cross-region load balancer, forwarding TCP 443 to both regions, and `*.catalyst.example.com` pointing at its address. `front_door_ip_address` is the address to allowlist in a firewall; it survives every failover.

:::tip Record the frontend ids now

Put the two ids from `gateway_frontend_ip_configurations` into `terraform.tfvars` as `primary_gateway_frontend_ip_configuration_id` and `secondary_gateway_frontend_ip_configuration_id`. With both set, this stack can still be applied while one region is down — which is exactly when you need to drain it.

:::

### Admit the front door's address in both regions

Traffic relayed by the front door arrives addressed to the front door's address, and AKS accepts it only if the gateway Service names that address. Add it to **both** regions' values files:

```yaml
gateway:
  envoy:
    service:
      annotations:
        # From make group-output front_door_ip_address.
        service.beta.kubernetes.io/azure-additional-public-ips: "<front_door_ip_address>"
```

and upgrade each release against its own cluster:

```bash
helm upgrade catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent -f catalyst-values.yaml --set join_token="${JOIN_TOKEN}"
```

The address never changes, so this is done once.

### Check the entry point

```bash
for r in catalyst-west catalyst-east; do
  printf '%s: ' "$r"
  curl -so /dev/null -m 10 -w '%{http_code}\n' "https://$r.$INGRESS_DOMAIN/diagrid/region/writable"
done
printf 'front door: '
curl -so /dev/null -m 10 -w '%{http_code}\n' "https://any-name.$INGRESS_DOMAIN/diagrid/region/writable"
```

Region 1 and the front door answer `200`. Region 2 gives no answer (`000`): a passive region's load balancer drops every connection, so it cannot be reached at its own name from outside. Anything else, see [Troubleshooting](#troubleshooting).

## 9. Create a project

Create projects in the group, never before it exists:

```bash
diagrid project create my-app --region-group my-group

diagrid project get my-app --output json | jq '.status.instances'
```

Each entry names the region it is placed in and its readiness. The project has one set of hostnames — `http-prj<id>.catalyst.example.com` and its `grpc-` equivalent — that resolve through the front door to whichever region is live. Applications hold a single endpoint and are unaware that a failover has occurred.

## 10. Verify the group

Run these checks before the group carries anything you care about.

**1. The control plane reports one writer and no mismatches.**

```bash
diagrid region group get my-group --output json | jq '.status'
```

Exactly one member accepts writes. A member that reports nothing has stopped heartbeating.

In `messages`, if you skipped the [shared PKI root](#optional-one-shared-pki-root), the one expected entry is `members do not share a PKI root (…)`. With a shared root the list is empty. `region catalyst-east does not resolve the KEK of region catalyst-west` means the two regions were given different `primary_encryption_key` values — correct it now, before any project writes a secret.

**2. Each region agrees with the control plane.** Re-run [the entry point check](#check-the-entry-point): the writer answers `200`, the other gives no answer, and the front door answers `200`.

**3. Replication lag is being scraped.** Every passive member exports [`cra_region_replication_lag_seconds`](https://docs.diagrid.io/operate/platform-operations/multi-region/metrics). That metric is your [RPO](https://docs.diagrid.io/operate/platform-operations/multi-region#recovery-objectives-rpo-and-rto), so confirm it reaches your monitoring and set an alert on it before you go live.

## 11. Rehearse a failover

Do this before the group serves production, not after.

A full round trip through [Azure failover and failback](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-failover) rebuilds **both** regions' databases, so plan the rehearsal as maintenance. Rehearse:

Record the wall-clock time from "we decided to promote" to "the first write succeeded in the new region". That number is your real [RTO](https://docs.diagrid.io/operate/platform-operations/multi-region#recovery-objectives-rpo-and-rto), and it is mostly decision time rather than machine time.

## Tear the group down

Follow this order. Deleting things in a different order fails partway through.

1. **Delete the projects, then the group.** `diagrid project delete`, then `diagrid region group delete my-group`. The member regions are not deleted with the group.
2. **Destroy the shared entry point**: `make group-destroy`, from the working directory that holds its state.
3. **`helm uninstall` in each region**, so the gateway releases the region's public address while the cluster still exists.
4. **End the replication.** Whichever region holds the replica, promote it with [`failover.sh promote`](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-failover#3-promote-the-replica).
5. **Destroy region 2, then region 1** — always in that order, even after a failover. Region 2 owns the network peering and records inside region 1's DNS zones, which region 1 cannot delete until they are gone.

## Troubleshooting

| Symptom | Cause and fix |
|---|---|
| `az vm list-usage` returns nothing, or an apply fails with `MissingSubscriptionRegistration` | The subscription is new and has no resource providers registered. Run `az provider register -n <namespace>` for `Microsoft.Compute`, `Microsoft.ContainerService`, `Microsoft.DBforPostgreSQL`, `Microsoft.Network`, `Microsoft.OperationalInsights` and `Microsoft.ManagedIdentity`, and wait until `az provider show -n <namespace> --query registrationState` reports `Registered`. |
| The apply fails on vCPU quota | Quota is counted per VM family as well as in total, per region. `az vm list-usage -l <region> -o table` shows both. If your node size's family has no row at all, it is not offered to your subscription and you need a different size. |
| The cluster create fails on zone placement | The node size is not available in the region's zones for your subscription. `az vm list-skus -l <region> --size <size> --zone -o table` returning nothing confirms it: set `availability_zones = []`, or to the zones it lists, in **both** regions. |
| `HA is disabled for region <region>` | The region does not offer zone-redundant PostgreSQL high availability to your subscription. Set `postgresql_high_availability = false` for that region. |
| `K8sVersionNotSupported` | The stack's `cluster_version` default has moved to Long-Term Support only. Pick a version shown as `KubernetesOfficial` in `az aks get-versions -l <region> -o table` in **both** regions, and set it in both. |
| `kubectl` fails with `executable kubelogin not found` | Install `kubelogin` with `az aks install-cli`, not Homebrew, and run `kubelogin convert-kubeconfig -l azurecli`. |
| `kubectl` fails with `User does not have access to the resource in Azure` | Your object id is not in `aks_admin_principal_ids`. Add it and re-apply. If it was already there, `kubelogin` is holding an old token: `rm -rf ~/.kube/cache/kubelogin`. |
| `helm install cert-manager` fails with `cannot unmarshal bool` | The pod label was passed with `--set`. It must be `--set-string`. |
| The Dapr scheduler never becomes ready and the agent restarts every five minutes, with `permission denied to start WAL sender` | The [`REPLICATION` grant](#let-the-database-user-stream-changes) was not made on region 1's database. |
| The second region's agent exits with `cannot execute CREATE TABLE in a read-only transaction` | The region joined before the group existed. It recovers on its own on the next restart once the group exists. |
| Region 1 gives no answer (`000`) at its own name | It is not a group member yet (`diagrid region group get my-group`), the certificate has not been issued, or the name has not propagated. A TLS error instead of `000` is the certificate. |
| Region 1 answers, the front door gives no answer | The gateway Service is missing [the front door's address](#admit-the-front-doors-address-in-both-regions). |

For problems after the group is running, see the [failover runbook's troubleshooting](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-failover#troubleshooting).

## Limitations

These are the limits of this Terraform, on top of the [limitations of a region group](https://docs.diagrid.io/operate/platform-operations/multi-region#limitations) itself:

- **The key encryption key is held in both clusters, not outside them.** Azure has no managed key encryption key provider in Catalyst yet. See [the key encryption key](#the-key-encryption-key).
- **Draining is all or nothing.** A cross-region load balancer has no traffic percentages, so draining a region removes it from the backend pool.
- **The front door must live in one of ten home regions** — Central US, East Asia, East US 2, North Europe, Southeast Asia, UK South, US Gov Virginia, West Europe, West US, or China North 2. The two member regions can be any pair.
- **It does not bound replication lag**, though the Catalyst agent measures it.
- **Both regions must be in one subscription.** The entry point stack uses a single provider for both members.

## Next steps

- [Azure failover and failback](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-failover) — The runbook: promote a region, move traffic, and rebuild the old one as a replica.
- [Multi-region high availability](https://docs.diagrid.io/operate/platform-operations/multi-region) — What a region group is, what its members share, and how a failover works on any cloud.
- [Production planning](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/production-planning) — Size, secure, and operate a self-managed region before it takes production traffic.