Azure failover and failback

This is the runbook for a two-region Catalyst region group on Azure, built with the Azure multi-region deployment guide. It covers recovering from the loss of a region, failing over for planned maintenance, failing back, and what to check when something does not behave.

A region is down right now?

Go straight to Unplanned failover. The rest of this page is context you can read afterwards.

Never clear postgresql_replicate_source_server_id to promote

Clearing it destroys the database you are failing over to. Promote with postgresql_promote_replica, and leave the source set — a promoted server keeps it for life. failover.sh promote refuses a plan that would replace a server for this reason. Do not work around the refusal.

How a failover works

Every failover on this page is the same two actions:

• Promote the replica. It becomes a writer, begins answering 200, and the front door starts sending traffic to it.
• Rebuild the other region as a replica: destroy its database and recreate it from the new writer. This is the only form of demotion available.

Both run through failover.sh, alongside the Makefile in each region's working directory. It plans through the same make targets you deploy with, shows you the plan, and refuses one that would destroy a database it was asked to promote. It decides nothing; you run it.

A promotion does not demote anything

Between promoting the new writer and rebuilding the old one, both regions' databases accept writes and both answer 200. Draining is the only thing keeping clients off the old one, so the drained region must stay drained until it has been rebuilt as a replica.

Unplanned failover

A region has been lost: its Kubernetes API, its gateway and its database are all unreachable.

Before you do anything:

• Traffic has already left the lost region. Its load balancer stopped reporting healthy, so the front door stopped sending it connections.
• Traffic has not arrived anywhere else. The surviving region is still a replica. Until you promote it, the group has no writer.

Recover

1. Set the promotion flag in the surviving region's terraform.tfvars, leaving the replication source where it is:

   postgresql_replicate_source_server_id = "<unchanged — do not clear this>"
   postgresql_promote_replica            = true

2. Promote, from that region's working directory:

   ./failover.sh promote --location eastus2 --ingress-endpoint $INGRESS_DOMAIN

   The script shows what it will promote, waits for confirmation, applies, and polls the region until it reports writable.

3. Drain the lost region before it can come back. In region-group/terraform.tfvars:

   primary_drained = true   # or secondary_drained, whichever region was lost

   make group-plan
   make group-apply

   This works while the lost region is down only if both primary_gateway_frontend_ip_configuration_id and secondary_gateway_frontend_ip_configuration_id are set, as the deployment guide's step 8 says.

What promoting costs you

Consequence	Detail
Data loss	Whatever had not replicated is gone: the replication lag at the moment the region failed.
Writes in flight fail	They fail at the database, and are retried by the client.
Jobs and reminders fire at least once	A job the old writer dispatched but had not recorded as done can fire again. Handlers must be idempotent.
The lost region returns as a second writer	Its database was never demoted, so when it returns it answers 200 and would receive traffic if it were not drained.

Once the lost region responds again, rebuild it with step 5 of the planned procedure, and keep it drained until that is done.

Planned failover

Five steps, in this order. Draining first stops writes before the database role moves.

1. Confirm which region is active

diagrid region group get my-group --output json | jq '.status.regions'

The control plane is the view to decide from. ./failover.sh status, run from the clone that holds the entry point state, also shows which regions are drained:

primary:   catalyst-west.catalyst.example.com   writable     in pool
secondary: catalyst-east.catalyst.example.com   no answer (a replica, or down: see 'diagrid region group get') in pool

From outside, a healthy replica and a lost region both give no answer, which is why status sends you to the control plane.

2. Drain the active region

In region-group/terraform.tfvars:

primary_drained = true

make group-plan
make group-apply

The region leaves the front door's backend pool. New requests stop reaching it within about 20 seconds; they time out rather than fail fast, because the only region left in the pool is still a replica.

3. Promote the replica

In the passive region's terraform.tfvars, set the promotion flag and leave the replication source alone:

postgresql_replicate_source_server_id = "<unchanged — do not clear this>"
postgresql_promote_replica            = true

Then, from that region's working directory:

./failover.sh promote --location eastus2 --ingress-endpoint $INGRESS_DOMAIN

The promotion itself takes about 20 seconds. If you normally pass other make variables to make apply in that directory, pass them after --, for example -- ENABLE_BASTION=true; otherwise the plan would undo what they configure, and the script refuses it. If the script refuses for any other reason, see Troubleshooting.

4. Let traffic follow

The promoted region starts answering 200 and the front door moves traffic to it, usually within a minute. No client re-resolves anything.

Leave the old region drained. Its database is still a writer.

5. Rebuild the old region as a replica

In the old region's terraform.tfvars, point it at the new writer and clear the promotion flag:

postgresql_replicate_source_server_id = "<the new writer's postgresql_server_id>"
postgresql_promote_replica            = false
# A replica cannot have high availability. Turn it back on after this region
# is next promoted.
postgresql_high_availability          = false

Then, from its working directory:

./failover.sh follow --location westus2 --ingress-endpoint $INGRESS_DOMAIN

This destroys that region's database and rebuilds it as a replica, which takes 10–15 minutes. The script asks for confirmation first. Flexible Server keeps the deleted server's automated backups for backup_retention_days, so its data stays recoverable for that window.

When it finishes, undrain the region. In region-group/terraform.tfvars:

primary_drained = false

make group-plan
make group-apply
./failover.sh status

status should show one writer and nothing drained. The group can now fail over in the other direction.

Fail back

Failing back is the planned failover again with the regions swapped: drain the current writer, promote the other region, rebuild the drained region as a replica, then undrain it.

After an unplanned failover, first rebuild the region that was lost as a replica — its data has diverged.

Each direction rebuilds one region's database, so a round trip rebuilds both. Let the new replica catch up before reversing direction: a promotion loses whatever has not yet replicated.

Recovery objectives (RPO and RTO)

The definitions are on the region group page. On Azure specifically:

Objective	On this architecture
RPO	The Flexible Server replication lag at the moment you promote. This Terraform does not bound it. Alert on cra_region_replication_lag_seconds, which every passive member exports.
RTO	Your promotion decision, plus a few minutes of machine time: the drain apply, a promotion of about 20 seconds, and the health probe turning. There is no DNS propagation to wait for. The rebuild afterwards is off the critical path.

Promotion is never automatic

No health probe promotes a database, in this Terraform or in Catalyst. The traffic half of a failover is automatic and the database half is not, and that is deliberate.

You can automate the rest yourself — an Azure Monitor availability test on each region, an alert rule, and something that runs failover.sh promote — if your RTO requires it. Read what is automatic and what is not first: on Flexible Server a promotion cannot be undone, so an unintended one costs a region rebuild.

Troubleshooting

Symptom	What to check
promote refuses because the plan replaces a server	postgresql_replicate_source_server_id has been cleared, or the SKU or storage changed in the same apply. Put the source back and make sizing changes separately, after the failover.
promote refuses because the plan promotes nothing	postgresql_promote_replica is not set — or, if this deployment has separate scheduler servers, scheduler_postgresql_promote_replicas.
promote refuses because a server is promoted with no replication source	The source was cleared. Put it back.
promote or follow refuses because the plan changes other resources	A make variable you normally pass to make apply is missing. Add it after --.
Azure refuses the apply with a replication-role error	A promoted server's role cannot be changed back. The source was cleared in a way Terraform did not plan as a replacement, or something is trying to change the role on an already-promoted server.
follow fails with ResourceNotFound (404) after creating the replica	Azure had not caught up with its own new server. The replica is usually fine: check that az postgres flexible-server replica list on the writer shows it Active. If it does, run terraform untaint azurerm_postgresql_flexible_server.postgresql in the terraform directory, then make plan: the only change left is the maintenance window, which is safe to apply.
Both regions report healthy, and traffic reaches the replica	Something set externalTrafficPolicy: Local on the gateway Service. With it, AKS ignores the probe path. Remove it.
The health probe never passes	The per-port annotations name the Service's own ports (port_443_… and port_8080_…), the 443 probe's protocol is https, and the path is /diagrid/region/writable on both.
A healthy region receives no traffic	It is still drained. ./failover.sh status shows which members are in the pool.
Traffic does not move after a successful promotion	Allow a minute for the load balancer health to catch up. If diagrid region group get my-group does not report the region writable, it was not promoted. If it does and the front door still times out, the gateway Service is missing the front door's address in azure-additional-public-ips.
The promoted region still reports not writable	Allow up to 60 seconds: the agent re-checks the database on its heartbeat.
Both regions report writable	Expected between a promotion and the rebuild that follows it. Outside that window, a rebuild was never done and the two databases are diverging.
Both regions answer 503 and neither is a replica	Expected briefly when a Catalyst agent restarts while the control plane is unreachable. It recovers when the agent reconnects.
The gateway's frontend was replaced	If the gateway Service was deleted and recreated, update the pinned *_gateway_frontend_ip_configuration_id from make group-output gateway_frontend_ip_configurations and re-apply the entry point stack.
The gateway Service has no external address	Both the azure-load-balancer-resource-group and azure-pip-name annotations must be set.
The group warns that the two regions do not resolve the same KEK	primary_encryption_key differs between the two values files. It must be the same 64 hex characters in both.
The passive region's scheduler never starts	wal_level must be logical on the writer, and the group needs PostgreSQL 16 or later. Setting wal_level restarts the writer.

Next steps

🌐

Azure multi-region deployment

Build the two-region group this runbook operates.

🌍

Multi-region high availability

What a region group is, what its members share, and how a failover works on any cloud.

📊

Metrics and alerts

The metrics that tell you your replication lag before a failover, not after.