# Azure failover and failback

This is the runbook for a two-region Catalyst [region group](https://docs.diagrid.io/operate/platform-operations/multi-region) on Azure, built with the [Azure multi-region deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-deployment) guide. It covers recovering from the loss of a region, failing over for planned maintenance, failing back, and what to check when something does not behave.

:::danger A region is down right now?

Go straight to [Unplanned failover](#unplanned-failover). The rest of this page is context you can read afterwards.

:::

:::danger Never clear `postgresql_replicate_source_server_id` to promote

Clearing it **destroys the database you are failing over to**. Promote with `postgresql_promote_replica`, and leave the source set — a promoted server keeps it for life. `failover.sh promote` refuses a plan that would replace a server for this reason. Do not work around the refusal.

:::

## How a failover works

Every failover on this page is the same two actions:

- **Promote** the replica. It becomes a writer, begins answering `200`, and the front door starts sending traffic to it.
- **Rebuild the other region as a replica**: destroy its database and recreate it from the new writer. This is the only form of demotion available.

Both run through `failover.sh`, alongside the `Makefile` in each region's working directory. It plans through the same `make` targets you deploy with, shows you the plan, and refuses one that would destroy a database it was asked to promote. It decides nothing; you run it.

:::warning A promotion does not demote anything

Between promoting the new writer and rebuilding the old one, **both regions' databases accept writes and both answer `200`**. Draining is the only thing keeping clients off the old one, so the drained region must stay drained until it has been rebuilt as a replica.

:::

## Unplanned failover

A region has been lost: its Kubernetes API, its gateway and its database are all unreachable.

Before you do anything:

- **Traffic has already left the lost region.** Its load balancer stopped reporting healthy, so the front door stopped sending it connections.
- **Traffic has not arrived anywhere else.** The surviving region is still a replica. Until you promote it, the group has no writer.

### Recover

1. **Set the promotion flag** in the surviving region's `terraform.tfvars`, leaving the replication source where it is:

   ```hcl
   postgresql_replicate_source_server_id = "<unchanged — do not clear this>"
   postgresql_promote_replica            = true
   ```

2. **Promote**, from that region's working directory:

   ```bash
   ./failover.sh promote --location eastus2 --ingress-endpoint $INGRESS_DOMAIN
   ```

   The script shows what it will promote, waits for confirmation, applies, and polls the region until it reports writable.

3. **Drain the lost region** before it can come back. In `region-group/terraform.tfvars`:

   ```hcl
   primary_drained = true   # or secondary_drained, whichever region was lost
   ```

   ```bash
   make group-plan
   make group-apply
   ```

   This works while the lost region is down only if both `primary_gateway_frontend_ip_configuration_id` and `secondary_gateway_frontend_ip_configuration_id` are set, as the deployment guide's step 8 says.

### What promoting costs you

| Consequence | Detail |
|---|---|
| **Data loss** | Whatever had not replicated is gone: the replication lag at the moment the region failed. |
| **Writes in flight fail** | They fail at the database, and are retried by the client. |
| **Jobs and reminders fire at least once** | A job the old writer dispatched but had not recorded as done can fire again. Handlers must be idempotent. |
| **The lost region returns as a second writer** | Its database was never demoted, so when it returns it answers `200` and would receive traffic if it were not drained. |

Once the lost region responds again, rebuild it with [step 5 of the planned procedure](#5-rebuild-the-old-region-as-a-replica), and keep it drained until that is done.

## Planned failover

Five steps, in this order. Draining first stops writes *before* the database role moves.

### 1. Confirm which region is active

```bash
diagrid region group get my-group --output json | jq '.status.regions'
```

The control plane is the view to decide from. `./failover.sh status`, run from the clone that holds the entry point state, also shows which regions are drained:

```
primary:   catalyst-west.catalyst.example.com   writable     in pool
secondary: catalyst-east.catalyst.example.com   no answer (a replica, or down: see 'diagrid region group get') in pool
```

From outside, a healthy replica and a lost region both give no answer, which is why `status` sends you to the control plane.

### 2. Drain the active region

In `region-group/terraform.tfvars`:

```hcl
primary_drained = true
```

```bash
make group-plan
make group-apply
```

The region leaves the front door's backend pool. New requests stop reaching it within about 20 seconds; they time out rather than fail fast, because the only region left in the pool is still a replica.

### 3. Promote the replica

In the passive region's `terraform.tfvars`, set the promotion flag and **leave the replication source alone**:

```hcl
postgresql_replicate_source_server_id = "<unchanged — do not clear this>"
postgresql_promote_replica            = true
```

Then, from that region's working directory:

```bash
./failover.sh promote --location eastus2 --ingress-endpoint $INGRESS_DOMAIN
```

The promotion itself takes about 20 seconds. If you normally pass other `make` variables to `make apply` in that directory, pass them after `--`, for example `-- ENABLE_BASTION=true`; otherwise the plan would undo what they configure, and the script refuses it. If the script refuses for any other reason, see [Troubleshooting](#troubleshooting).

### 4. Let traffic follow

The promoted region starts answering `200` and the front door moves traffic to it, usually within a minute. No client re-resolves anything.

**Leave the old region drained.** Its database is still a writer.

### 5. Rebuild the old region as a replica

In the old region's `terraform.tfvars`, point it at the new writer and clear the promotion flag:

```hcl
postgresql_replicate_source_server_id = "<the new writer's postgresql_server_id>"
postgresql_promote_replica            = false
# A replica cannot have high availability. Turn it back on after this region
# is next promoted.
postgresql_high_availability          = false
```

Then, from its working directory:

```bash
./failover.sh follow --location westus2 --ingress-endpoint $INGRESS_DOMAIN
```

This **destroys** that region's database and rebuilds it as a replica, which takes 10–15 minutes. The script asks for confirmation first. Flexible Server keeps the deleted server's automated backups for `backup_retention_days`, so its data stays recoverable for that window.

When it finishes, undrain the region. In `region-group/terraform.tfvars`:

```hcl
primary_drained = false
```

```bash
make group-plan
make group-apply
./failover.sh status
```

`status` should show one writer and nothing drained. The group can now fail over in the other direction.

## Fail back

Failing back is the planned failover again with the regions swapped: drain the current writer, promote the other region, rebuild the drained region as a replica, then undrain it.

After an unplanned failover, first rebuild the region that was lost as a replica — its data has diverged.

Each direction rebuilds one region's database, so a round trip rebuilds both. Let the new replica catch up before reversing direction: a promotion loses whatever has not yet replicated.

## Recovery objectives (RPO and RTO)

[The definitions are on the region group page](https://docs.diagrid.io/operate/platform-operations/multi-region#recovery-objectives-rpo-and-rto). On Azure specifically:

| Objective | On this architecture |
|---|---|
| **RPO** | The Flexible Server replication lag at the moment you promote. This Terraform does not bound it. Alert on [`cra_region_replication_lag_seconds`](https://docs.diagrid.io/operate/platform-operations/multi-region/metrics), which every passive member exports. |
| **RTO** | Your promotion decision, plus a few minutes of machine time: the drain apply, a promotion of about 20 seconds, and the health probe turning. There is no DNS propagation to wait for. The rebuild afterwards is off the critical path. |

## Promotion is never automatic

No health probe promotes a database, in this Terraform or in Catalyst. The traffic half of a failover is automatic and the database half is not, and [that is deliberate](https://docs.diagrid.io/operate/platform-operations/multi-region#failover-what-is-automatic-and-what-is-not).

You can automate the rest yourself — an Azure Monitor availability test on each region, an alert rule, and something that runs `failover.sh promote` — if your RTO requires it. Read [what is automatic and what is not](https://docs.diagrid.io/operate/platform-operations/multi-region#failover-what-is-automatic-and-what-is-not) first: on Flexible Server a promotion cannot be undone, so an unintended one costs a region rebuild.

## Troubleshooting

| Symptom | What to check |
|---|---|
| `promote` refuses because the plan replaces a server | `postgresql_replicate_source_server_id` has been cleared, or the SKU or storage changed in the same apply. Put the source back and make sizing changes separately, after the failover. |
| `promote` refuses because the plan promotes nothing | `postgresql_promote_replica` is not set — or, if this deployment has separate scheduler servers, `scheduler_postgresql_promote_replicas`. |
| `promote` refuses because a server is promoted with no replication source | The source was cleared. Put it back. |
| `promote` or `follow` refuses because the plan changes other resources | A `make` variable you normally pass to `make apply` is missing. Add it after `--`. |
| Azure refuses the apply with a replication-role error | A promoted server's role cannot be changed back. The source was cleared in a way Terraform did not plan as a replacement, or something is trying to change the role on an already-promoted server. |
| `follow` fails with `ResourceNotFound` (404) *after* creating the replica | Azure had not caught up with its own new server. The replica is usually fine: check that `az postgres flexible-server replica list` on the writer shows it `Active`. If it does, run `terraform untaint azurerm_postgresql_flexible_server.postgresql` in the `terraform` directory, then `make plan`: the only change left is the maintenance window, which is safe to apply. |
| Both regions report healthy, and traffic reaches the replica | Something set `externalTrafficPolicy: Local` on the gateway Service. With it, AKS ignores the probe path. Remove it. |
| The health probe never passes | The per-port annotations name the Service's own ports (`port_443_…` and `port_8080_…`), the 443 probe's protocol is `https`, and the path is `/diagrid/region/writable` on both. |
| A healthy region receives no traffic | It is still drained. `./failover.sh status` shows which members are in the pool. |
| Traffic does not move after a successful promotion | Allow a minute for the load balancer health to catch up. If `diagrid region group get my-group` does not report the region writable, it was not promoted. If it does and the front door still times out, the gateway Service is missing the front door's address in `azure-additional-public-ips`. |
| The promoted region still reports not writable | Allow up to 60 seconds: the agent re-checks the database on its heartbeat. |
| Both regions report `writable` | Expected between a promotion and the rebuild that follows it. Outside that window, a rebuild was never done and the two databases are diverging. |
| Both regions answer `503` and neither is a replica | Expected briefly when a Catalyst agent restarts while the control plane is unreachable. It recovers when the agent reconnects. |
| The gateway's frontend was replaced | If the gateway Service was deleted and recreated, update the pinned `*_gateway_frontend_ip_configuration_id` from `make group-output gateway_frontend_ip_configurations` and re-apply the entry point stack. |
| The gateway Service has no external address | Both the `azure-load-balancer-resource-group` and `azure-pip-name` annotations must be set. |
| The group warns that the two regions do not resolve the same KEK | `primary_encryption_key` differs between the two values files. It must be the same 64 hex characters in both. |
| The passive region's scheduler never starts | `wal_level` must be `logical` on the **writer**, and the group needs PostgreSQL 16 or later. Setting `wal_level` restarts the writer. |

## Next steps

- [Azure multi-region deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-deployment) — Build the two-region group this runbook operates.
- [Multi-region high availability](https://docs.diagrid.io/operate/platform-operations/multi-region) — What a region group is, what its members share, and how a failover works on any cloud.
- [Metrics and alerts](https://docs.diagrid.io/operate/platform-operations/multi-region/metrics) — The metrics that tell you your replication lag before a failover, not after.