AWS failover and failback
This is the runbook for a two-region Catalyst region group on AWS, built with the AWS multi-region deployment guide. It covers recovering from the loss of a region, failing over for planned maintenance, failing back, and what to check when something does not behave.
Go straight to Unplanned failover. The rest of this page is context you can read afterwards.
How a failover works
Every failover on this page is the same two actions, in one order or the other:
- Promote the replica. It becomes a standalone primary, begins answering
200, and the accelerator starts sending traffic to it. - Rebuild the other region as a replica: destroy its databases and rebuild them from the new primary. This is the demotion, and it is the only form of demotion available.
Both run through failover.sh, alongside the Makefile in each region's working directory. It plans through the same make target used to deploy, reads that plan, and refuses to apply one that would destroy a database it was asked to promote. It never calls the RDS API directly, so the Terraform state remains accurate and the next ordinary make plan is clean. The script decides nothing; you run it.
RDS promotes a replica; it cannot demote a primary. Between promoting the new primary and rebuilding the old one, both regions' databases accept writes and both answer 200. The traffic dial is the only thing keeping clients off the old one, which is why the drained region must stay drained until it has been rebuilt as a replica.
Unplanned failover
A region has been lost. Its Kubernetes API, its gateway and its database are all unreachable, and nothing in it responds until AWS restores it.
Three things are already true before you take any action:
- Traffic has left the lost region. Its load balancer stopped reporting healthy targets, so the accelerator stopped sending it connections. No decision was required.
- Traffic has not arrived anywhere else. The surviving region is still a replica, still answering
503, and still unhealthy to the accelerator. Until you promote it, the group has no primary and no healthy endpoint. - There is nothing to drain, and nothing to rebuild yet. Draining exists to stop writes before the database role moves, and here it moved first. Rebuilding a region as a replica requires that region to respond.
Recover
Recovery is the promotion from the planned procedure, run on its own.
-
Clear both replication variables in the surviving region's
terraform.tfvars:postgresql_replicate_source_db_arn = ""scheduler_postgresql_replicate_source_db_arns = {} -
Promote, from that region's working directory:
./failover.sh promote --region us-east-1 --ingress-endpoint $INGRESS_DOMAINThe script lists what it is about to promote, waits for confirmation, applies the plan it displayed, and then polls the region until it reports writable.
-
Set the lost region's traffic dial to
0, while it is still down. Inregion-group/terraform.tfvars:primary_traffic_dial_percentage = 0make group-planmake group-applyDo this before the lost region can receive any traffic. This is what
primary_gateway_lb_arnandsecondary_gateway_lb_arnare for: with both set, the entry point stack reads nothing from either member region and can be applied while one of them is gone. Without them, each load balancer is discovered by tag, and the lost region cannot answer that lookup until it returns — by which point it may already be receiving traffic.
What promoting costs you
| Consequence | Detail |
|---|---|
| Data loss | Whatever had not replicated is gone. The loss equals the replication lag at the moment the region failed. |
| Writes in flight fail | They fail at the database, and are retried by the client. |
| Jobs and reminders fire at least once | A job the old primary had claimed and dispatched but not recorded as done can fire again in the new one. Catalyst is not notified that a promotion has occurred, so handlers must be idempotent. |
| The lost region returns as a second primary | Its database was never demoted, so when it returns it answers 200, its targets become healthy, and the accelerator sends it traffic alongside the promoted region. Two primaries, diverging data. |
Once the lost region responds again, rebuild it as a replica with step 5 of the planned procedure. Keep its dial at 0 until that is done.
Planned failover
Five steps, in this order. Draining first ensures that writes stop before the database role moves, rather than as a consequence of it.
1. Confirm which region is active
./failover.sh status
primary: catalyst-west.catalyst.example.com writable traffic dial 100%
secondary: catalyst-east.catalyst.example.com replica traffic dial 100%
This reads the entry point stack's state, so run it from the clone that holds it. diagrid regiongroup get my-group is the control plane's view of the same information.
2. Drain the active region
In region-group/terraform.tfvars, set that region's dial to zero and apply:
primary_traffic_dial_percentage = 0
make group-plan
make group-apply
The accelerator stops sending that region new connections while its load balancer is still healthy. New work fails fast instead of reaching a database that is about to change.
3. Promote the replica
In the passive region's terraform.tfvars, clear both replication variables:
postgresql_replicate_source_db_arn = ""
scheduler_postgresql_replicate_source_db_arns = {}
Then, from that region's working directory:
./failover.sh promote --region us-east-1 --ingress-endpoint $INGRESS_DOMAIN
Removing the replication source promotes the instances in place. The script lists what it is about to promote, waits for confirmation, applies the plan it displayed, and then polls the region until it reports writable.
Pass any make variable you normally pass to make apply in that directory after a --, for example -- ENABLE_BASTION=false. If one is omitted, the plan proposes undoing whatever that variable configures, and the script refuses the plan rather than applying it.
promote stops when the plan replaces a database instance instead of updating it, because replacement destroys the data being failed over to. db_name and username force replacement when they change, and they go from unset to their literal values at this step, so a refusal here means those two values differ between the regions. Correct the mismatched variable rather than working around the check.
A promotion also fails while an instance is in backing-up. That is transient; run it again.
4. Let traffic follow
The promoted region starts answering 200, its load balancer's targets become healthy, and the accelerator moves traffic. This takes under a minute, and no client re-resolves anything.
Leave the old region drained. Its database is still a primary and still answers 200. Undraining it now would split traffic across two regions that both accept writes.
5. Rebuild the old region as a replica
This is the demotion. In its terraform.tfvars, point both replication variables at the new primary's postgresql_arn and scheduler_postgresql_arns outputs, then from its working directory:
./failover.sh follow --region us-west-2 --ingress-endpoint $INGRESS_DOMAIN
This destroys that region's databases and rebuilds them as replicas. Their contents are stale by this point, since the new primary holds the current state, but the script still requires confirmation, and postgresql_deletion_protection must be false for the apply to run. postgresql_skip_final_snapshot determines whether a snapshot is kept first.
Once it is a replica it answers 503 again. Set its dial back to 100 and apply the entry point stack: the group is symmetric again and can fail over in the other direction.
./failover.sh status
Confirms both halves — one primary, and no dial left at zero.
Fail back
Failing back is the planned failover procedure again with the regions swapped: drain the current primary, promote the other region's replica, rebuild the drained region as a replica, then undrain it. Catalyst has no asymmetry here and no separate mechanism: whichever region holds the primary is the active one.
After an unplanned failover there is one additional step first. The region that was lost returns holding a primary whose data has diverged, so rebuild it as a replica before you fail back to it.
Each direction rebuilds one region's databases from scratch, so a round trip rebuilds both. Wait for the new replica to catch up before reversing direction: a promotion loses whatever has not yet replicated.
Recovery objectives (RPO and RTO)
The definitions are on the region group page. On AWS specifically:
| Objective | On this architecture |
|---|---|
| RPO | The RDS cross-region replication lag at the moment you promote. This Terraform does not bound it. Alert on cra_region_replication_lag_seconds, which every passive member exports. |
| RTO | Your promotion decision, plus the traffic move. The traffic move is the smaller half: the health check changes within about 20 seconds of the write status changing, and the accelerator's addresses do not change, so there is no DNS propagation to wait for. Catalyst's own contribution is the time from promotion to the first successful write. |
Promotion is never automatic
No health check promotes a database, in this Terraform or in Catalyst. The traffic half of a failover is automatic and the database half is not, and that is deliberate rather than unfinished.
You can build the remaining half on AWS — a Route 53 health check on each region's own hostname, a CloudWatch alarm in the surviving region, and a component that runs the promotion — and you should if your RTO requires it. Read what is automatic and what is not before you do. One of the reasons there bites harder on RDS than elsewhere: you recover from an unintended promotion only by rebuilding a region as a replica, which destroys everything written to it since.
Troubleshooting
| Symptom | What to check |
|---|---|
| Both regions time out, or the health check never passes | healthcheck-port names the same port as httpsNodePort, and the health check protocol is HTTPS rather than HTTP. |
| A healthy region receives no traffic | A traffic_dial_percentage left at 0 from an earlier planned failover. ./failover.sh status shows both dials alongside which region is active. |
| Traffic does not move after a successful promotion | ./failover.sh status queries each region by its own name. If the promoted region reports writable, the target group health has not caught up yet; allow a minute. If it still reports replica, the promotion did not take effect. |
Both regions report writable | Expected between a promotion and the rebuild that follows it — the drained region's dial is what keeps clients off the old primary. Outside that window it means a rebuild was never performed and the two databases are diverging. |
| The gateway load balancer was replaced | The accelerator holds its ARN. Re-apply the entry point stack to pick up the new one, and if that region's ARN is pinned in primary_gateway_lb_arn or secondary_gateway_lb_arn, update it first: a pinned ARN is not rediscovered. |
A pod logs an AccessDeniedException from KMS | The service account is not assuming the key encryption key (KEK) role. Check that global.serviceAccount.annotations carries eks.amazonaws.com/role-arn, and that the service account names in kek_kms_service_account_subjects match the ones the chart created. They are derived from the Helm release name, so installing under a name other than catalyst changes them. |
| The group warns that the two regions do not resolve the same KEK | Compare aws_kms_key_id in both values files. It must be the key id, identical in both. The region-local ARNs are two different strings for the same key, and the comparison reads them as two keys. |
Both regions answer 503 and neither is a replica | Expected briefly when a Catalyst agent restarts during a control plane outage. Group membership reaches the agent over the control plane watch and is held in memory, so a restarted agent sees itself as standalone until the watch is redelivered. |
| The promoted region still reports not writable | The agent re-observes the database on its heartbeat cadence and holds the previous answer through one failed check. Allow up to 60 seconds before treating it as stuck. |
