# AWS failover and failback

This is the runbook for a two-region Catalyst [region group](https://docs.diagrid.io/operate/platform-operations/multi-region) on AWS, built with the [AWS multi-region deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-deployment) guide. It covers what to do when a region goes down, how to fail over for planned maintenance, how to fail back, and what to check when something goes wrong.

:::danger Is a region down right now?

Go straight to [Unplanned failover](#unplanned-failover). You can read the rest of this page later.

:::

## How a failover works

Every failover on this page uses the same two actions:

- **Promote** the replica. It becomes a standalone primary, starts returning `200`, and the accelerator starts sending it traffic.
- **Rebuild the other region as a replica.** Delete its databases and rebuild them from the new primary. This is the only way to demote a region.

You run both with `failover.sh`, which sits next to the `Makefile` in each region's working directory. The script plans with the same `make` target you used to deploy, checks the plan, and refuses to apply a plan that would delete a database you asked it to promote. It never calls the RDS API directly, so your Terraform state stays accurate and your next `make plan` is clean. The script doesn't make any decisions. You decide when to run it.

```mermaid
---
title: Planned failover, step by step
---
flowchart TD
  START((Start)):::startNode
  DRAIN("1. Drain the active region<br/>traffic dial to 0")
  PROMOTE("2. Promote the replica<br/>failover.sh promote")
  TRAFFIC("3. Traffic follows<br/>the health check turns")
  REBUILD("4. Rebuild the old region as a replica<br/>failover.sh follow")
  UNDRAIN("5. Undrain it<br/>traffic dial back to 100")
  END((End)):::endNode
  START-->DRAIN
  DRAIN-->PROMOTE
  PROMOTE-->TRAFFIC
  TRAFFIC-->REBUILD
  REBUILD-->UNDRAIN
  UNDRAIN-->END

  classDef startNode stroke:#22613f,stroke-width:3px
  classDef endNode stroke:#8b1a1a,stroke-width:3px
```

**Promoting doesn't demote anything.** RDS can promote a replica, but it can't turn a primary back into a replica. So between promoting the new primary and rebuilding the old one, **both regions' databases accept writes, and both return `200`**. Only the traffic dial keeps clients away from the old one. That's why the drained region must stay drained until you've rebuilt it as a replica.

## Unplanned failover

A region is down. Its Kubernetes API, gateway, and database are all unreachable, and nothing in it will respond until AWS brings it back.

Before you do anything, three things are already true:

- **Traffic has left the lost region.** Its load balancer stopped reporting healthy targets, so the accelerator stopped sending it connections. You didn't have to do anything.
- **Traffic hasn't gone anywhere else.** The other region is still a replica. It still returns `503`, so the accelerator still treats it as unhealthy. Until you promote it, the group has no primary and no healthy endpoint.
- **There's nothing to drain, and nothing to rebuild yet.** You drain to stop writes before the database role moves, but here the region is already gone. And you can only rebuild a region as a replica once it responds again.

### Recover

To recover, you run the promotion step from the planned procedure on its own.

1. **Clear the replication source** in the surviving region's `terraform.tfvars`:

   ```hcl
   postgresql_replicate_source_db_arn = ""
   ```

2. **Promote the replica**, from that region's working directory:

   ```bash
   ./failover.sh promote --region us-east-1 --ingress-endpoint $INGRESS_DOMAIN
   ```

   The script shows what it's about to promote and waits for you to confirm. Then it applies the plan it showed you and checks the region until it reports that it accepts writes.

3. **Set the lost region's traffic dial to `0`** while it's still down. In `region-group/terraform.tfvars`:

   ```hcl
   primary_traffic_dial_percentage = 0
   ```

   ```bash
   make group-plan
   make group-apply
   ```

   Do this before the lost region can get any traffic. This is why you set `primary_gateway_lb_arn` and `secondary_gateway_lb_arn`: with both set, the entry point stack doesn't need to contact either region, so you can apply it while one of them is down. Without them, the stack finds each load balancer by tag, and the lost region can't answer that lookup until it comes back. By then it might already be getting traffic.

### What promoting costs you

| Effect | Detail |
|---|---|
| **Data loss** | Whatever hadn't replicated is lost. That's the replication lag at the moment the region went down. |
| **Writes in progress fail** | They fail at the database, and the client retries them. |
| **Jobs and reminders run at least once** | If the old primary started a job but hadn't recorded it as done, the new primary can run it again. Catalyst isn't told that a promotion happened, so your handlers must be idempotent. |
| **The lost region comes back as a second primary** | Nothing demoted its database. When it comes back, it returns `200`, its targets become healthy, and the accelerator sends it traffic alongside the promoted region. You then have two primaries whose data is drifting apart. |

Once the lost region responds again, rebuild it as a replica with [step 5 of the planned procedure](#5-rebuild-the-old-region-as-a-replica). Keep its traffic dial at `0` until you're done.

## Planned failover

Follow these five steps in order. You drain first so writes stop *before* the database role moves.

### 1. Confirm which region is active

```bash
./failover.sh status
```

```
primary:   catalyst-west.catalyst.example.com            writable     traffic dial 100%
secondary: catalyst-east.catalyst.example.com            replica      traffic dial 100%
```

This reads the entry point stack's state, so run it from the clone that holds that state. `diagrid region group get my-group` shows the control plane's view of the same thing.

### 2. Drain the active region

In `region-group/terraform.tfvars`, set that region's dial to `0`, and apply:

```hcl
primary_traffic_dial_percentage = 0
```

```bash
make group-plan
make group-apply
```

The accelerator stops sending new connections to that region, even though its load balancer is still healthy. New requests fail quickly instead of reaching a database that's about to change.

### 3. Promote the replica

In the passive region's `terraform.tfvars`, clear the replication source:

```hcl
postgresql_replicate_source_db_arn = ""
```

Then, from that region's working directory:

```bash
./failover.sh promote --region us-east-1 --ingress-endpoint $INGRESS_DOMAIN
```

Clearing the replication source promotes the database in place. The script shows what it's about to promote and waits for you to confirm. Then it applies the plan it showed you and checks the region until it reports that it accepts writes.

If you usually pass other `make` variables to `make apply` in that directory, pass them after `--`, for example `-- ENABLE_BASTION=false`. If you leave one out, the plan tries to undo whatever that variable sets up, and the script refuses the plan.

**If the script refuses to apply**, the plan **replaces** a database instead of updating it. Replacing it would delete the data you're failing over to, so the script stops. Changing `db_name` or `username` forces a replacement, and at this step they go from unset to their actual values. So a refusal here means those two values are different in the two regions. Fix the variable that doesn't match. Don't work around the check.

A promotion also fails while the instance is in the `backing-up` state. That's temporary, so run it again.

### 4. Let traffic follow

The promoted region starts returning `200`, its load balancer's targets become healthy, and the accelerator moves traffic to it. This takes less than a minute, and clients don't need to look up any new addresses.

**Keep the old region drained.** Its database is still a primary and still returns `200`. If you undrain it now, traffic splits across two regions that both accept writes.

### 5. Rebuild the old region as a replica

This is the demotion. In the old region's `terraform.tfvars`, set `postgresql_replicate_source_db_arn` to the new primary's `postgresql_arn` output. Then, from its working directory, run:

```bash
./failover.sh follow --region us-west-2 --ingress-endpoint $INGRESS_DOMAIN
```

This **deletes** the region's database and rebuilds it as a replica. The old data is out of date by now, because the new primary has the current state, but the script still asks you to confirm. `postgresql_skip_final_snapshot` controls whether it keeps a snapshot first.

`follow` tells Terraform directly to replace the instance, instead of relying on the plan to do it. This matters from the second failover on. The AWS provider only creates a new instance for `replicate_source_db` if that setting wasn't in the previous state. So a database that has been a replica before would otherwise plan as an in-place update, which AWS can't do, because there's no API to demote a database.

**If `follow` refuses because of deletion protection**, the instance it's about to delete still has `postgresql_deletion_protection = true`. You can't clear that flag in the same apply as the rebuild. Pointing an instance at a replication source replaces it, so Terraform tries to delete it before it applies the new setting, and RDS refuses every time you retry. Clear the flag in a separate apply:

1. Leave `postgresql_replicate_source_db_arn` empty, and set `postgresql_deletion_protection = false`.
2. Run `make apply REGION=us-west-2 REGION_INGRESS_ENDPOINT=$INGRESS_DOMAIN`. This updates the instance in place and deletes nothing.
3. Set the replication source again, and run `follow` again.

That's why the [deployment guide](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-deployment#3-deploy-the-first-regions-aws-infrastructure) sets the flag to `false` from the start.

Once the old region is a replica, it returns `503` again. Set its dial back to `100` and apply the entry point stack. The group is back to normal and can fail over in either direction.

```bash
./failover.sh status
```

This confirms both parts: one primary, and no traffic dial left at `0`.

## Fail back

To fail back, run the planned failover again with the regions swapped: drain the current primary, promote the other region's replica, rebuild the drained region as a replica, and undrain it. There's no separate failback process, and neither region is special. Whichever region holds the primary is the active one.

After an unplanned failover, there's one extra step first. The region that went down comes back with a primary whose data has drifted, so rebuild it as a replica before you fail back to it.

Each failover rebuilds one region's databases from scratch, so a failover and failback rebuild both. Wait for the new replica to catch up before you fail back, because a promotion loses whatever hasn't replicated yet.

## Recovery objectives (RPO and RTO)

See the [region group page](https://docs.diagrid.io/operate/platform-operations/multi-region#recovery-objectives-rpo-and-rto) for what RPO and RTO mean for a group. On AWS:

| Objective | On this setup |
|---|---|
| **RPO** | The RDS cross-region replication lag when you promote. This Terraform doesn't cap it. Set an alert on [`cra_region_replication_lag_seconds`](https://docs.diagrid.io/operate/platform-operations/multi-region/metrics), which every passive member exports. |
| **RTO** | The time you take to decide to promote, plus the time to move traffic. Moving traffic is the smaller part. The health check changes within about 20 seconds of the database changing, and the accelerator's addresses stay the same, so there's no DNS change to wait for. Catalyst itself adds the time from the promotion to the first successful write. |

## Promotion is never automatic

No health check promotes a database, in this Terraform or in Catalyst. Moving traffic is automatic, but promoting the database isn't. [That's on purpose](https://docs.diagrid.io/operate/platform-operations/multi-region#failover-what-is-automatic-and-what-is-not).

You can automate promotion on AWS, and you should if your RTO needs it. For example, use a Route 53 health check on each region's own hostname, a CloudWatch alarm in the surviving region, and something that runs the promotion. Read [what is automatic and what is not](https://docs.diagrid.io/operate/platform-operations/multi-region#failover-what-is-automatic-and-what-is-not) before you do. On RDS, an unneeded promotion is especially costly: the only fix is to rebuild a region as a replica, which deletes everything written to it since the promotion.

## Troubleshooting

| Symptom | What to check |
|---|---|
| Both regions time out, or the health check never passes | Check that `healthcheck-port` is the same port as `httpsNodePort`, and that the health check protocol is `HTTPS`, not `HTTP`. |
| A healthy region gets no traffic | A `traffic_dial_percentage` might still be at `0` from an earlier planned failover. `./failover.sh status` shows both dials and which region is active. |
| Traffic doesn't move after a successful promotion | `./failover.sh status` checks each region by its own name. If the promoted region shows `writable`, the target group health hasn't caught up yet, so wait a minute. If it still shows `replica`, the promotion didn't take effect. |
| Both regions show `writable` | This is expected between a promotion and the rebuild after it. The drained region's dial keeps clients away from the old primary. At any other time, it means a rebuild never happened and the two databases are drifting apart. |
| The gateway load balancer was replaced | The accelerator stores the load balancer's ARN. Re-apply the entry point stack to pick up the new one. If that region's ARN is set in `primary_gateway_lb_arn` or `secondary_gateway_lb_arn`, update it first, because the stack doesn't look up a set ARN again. |
| A pod logs an `AccessDeniedException` from KMS | The service account isn't using the key encryption key (KEK) role. Check that `global.serviceAccount.annotations` includes `eks.amazonaws.com/role-arn`, and that the service account names in `kek_kms_service_account_subjects` match the ones the chart created. The names come from the Helm release name, so if you installed with a release name other than `catalyst`, they're different. |
| The group warns that the two regions don't resolve the same KEK | Compare `aws_kms_key_id` in both values files. It must be the key ID, and the same in both. The region-specific ARNs are different strings for the same key, so the check treats them as two keys. |
| Both regions return `503`, and neither is a replica | This can happen for a short time when a Catalyst agent restarts during a control plane outage. The agent gets its group membership from the control plane and keeps it in memory, so after a restart it thinks it's standalone until the control plane sends the membership again. |
| The promoted region still shows it doesn't accept writes | The agent checks the database on each heartbeat, and keeps its previous answer through one failed check. Wait up to 60 seconds before you treat it as stuck. |

## Next steps

- [AWS multi-region deployment](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-deployment) — Build the two-region group this runbook covers.
- [Multi-region high availability](https://docs.diagrid.io/operate/platform-operations/multi-region) — What a region group is, what its members share, and how a failover works on any cloud.
- [Metrics and alerts](https://docs.diagrid.io/operate/platform-operations/multi-region/metrics) — Track your replication lag before a failover, not after.