Multi-region high availability

Catalyst keeps your workflows and agents running when a pod restarts, a node fails, or an availability zone goes down. None of that helps when the whole cloud region goes down: the cluster, its database, and its gateway are all in the same place.

A region group solves that. It connects several Catalyst regions to one replicated database, so a project can survive the loss of an entire region. A project created in the group is placed in every member region and keeps one set of hostnames. Your applications hold a single endpoint and never learn which region is serving them.

A group is active/passive. One member runs against the database that accepts writes — the active member — and it serves all traffic. The others run against read-only replicas. They stay deployed and keep their applications running, but they take over only once their database is promoted.

At a glance

Property	What to expect
Model	Active/passive. Exactly one member accepts writes at a time and serves all traffic.
Traffic failover	Automatic. The load balancer follows the health check, within about 20 seconds.
Database failover	Manual. You promote the replica — and that is deliberate.
RPO — how much data you lose	The replication lag at the moment you promote. Replication is asynchronous, so Catalyst does not bound it. Every passive member exports cra_region_replication_lag_seconds, so you know your exposure in advance.
RTO — how long recovery takes	Mostly how long you take to decide to promote. The traffic move is the smaller half, at about 20 seconds.
What fails over	Catalyst's own state: projects, components, secrets, workflow history, jobs and reminders.
What does not	Anything regional your project depends on — a pub/sub broker, a rate-limit store. Those must be multi-region in their own right.
What your applications see	One endpoint and one set of hostnames, before and after a failover. No redeploy, no configuration change.
Cost	Roughly double a single region. A passive member is a full region and serves no traffic.
Available on	Self-managed BYOC regions today.

Plan the group before its regions hold anything

Every member must be a region of the same type, and empty when it joins. You cannot move an existing project into a group, and you cannot move a project out of one. Today the documented path is self-managed BYOC regions, where you own the database, the keys, and the network that make a group work. For Dedicated Cloud or managed BYOC regions, contact Diagrid.

When to use a region group

Your goal	Use this
Keep a project running through the loss of a whole cloud region	A region group
Survive the loss of a node or an availability zone inside one region	High availability in that region
Serve users closer to them, or add throughput	Independent regions
Isolate environments, business units, or data residency domains	Independent regions

A region group is a disaster recovery mechanism, not a performance one. A passive member serves no traffic, so a group adds no throughput and shortens no round trip. It also does not replace high availability inside a region — most deployments want both.

What survives what

Each column adds to the one before it. A region group is the last column, and it is the only one that covers the loss of a whole cloud region.

Failure	Single region, default install	Single region with HA	Region group
A pod restarts or crashes	Survives	Survives	Survives
A Kubernetes node is lost	Survives, with a gap in serving while single-replica pods reschedule	Survives	Survives
An availability zone is lost	Outage until the zone returns	Survives	Survives
A cloud region is lost	Outage until the region returns	Outage until the region returns	Survives, once you promote the replica
A dependency of your own is lost — a pub/sub broker, a rate-limit store	Outage	Outage	Outage. A group fails over Catalyst's state, not your broker

Workflows and agents are durable in every column: they are recorded as they run, so they resume wherever the platform comes back rather than starting over. What the columns differ on is how long the platform is unavailable in between.

"Single region with HA" means gateway.ha.enabled: true and the external PostgreSQL and Kafka deployed across availability zones, as Production planning describes. A region group does not replace that. A member region without HA of its own still fails over for a zone outage rather than riding through it, which costs you a promotion and whatever had not replicated. Most production deployments want both.

Recovery starts with a decision you make, not one the platform makes for you. See Failover: what is automatic and what is not.

Budget for a second full region

A passive member is a full region. It runs the same data plane at the same size and serves no traffic, so a two-region group roughly doubles the infrastructure cost of a single region.

Architecture

Every member of a group publishes a health check endpoint on its gateway, at /diagrid/region/writable. It answers:

Response	Meaning
200	This region's database accepts writes. It is the active member.
503	This region runs against a read-only replica. It is a passive member.
404	This region is not in a group.

You put a global load balancer in front of the member regions, point its health check at that path, and it sends traffic to whichever member answers 200.

The database is therefore the only authority on which region is active. Catalyst observes what the database reports and publishes it. It never changes it.

What a failover does to running work

Your workflows, agents, jobs and reminders live in the group's replicated database, so a passive member already holds them. When it is promoted it continues them from the history it has replicated, and what that history contains at that moment determines everything below.

• Workflows and agents continue where the replica had them. The promoted member replays each one from its recorded history and carries on. Nothing is lost that had replicated, and nothing has to be resubmitted.
• Whatever had not replicated is gone. Replication is asynchronous, so a workflow that advanced in the seconds before the failover resumes from its last replicated step. The activities after that step run again.
• Activities, jobs and reminders run at least once. A job the old member had dispatched but not recorded as complete fires again in the promoted one. Catalyst is never told that a promotion has occurred, so nothing suppresses the duplicate. Handlers must be idempotent — this is the one thing a region group asks of your application code.
• Calls in flight fail. They reach a database that no longer accepts writes, or a region traffic has already left. The SDKs retry them, and they succeed against the promoted member once it is writable.
• Resources created mid-failover stay pending. A project, component or App ID created while the group has no writer is provisioned once writes are accepted again.
• Signed workflow history needs one PKI root for the whole group. If a project signs its workflow history and the members do not share a root CA, a workflow resumed in another member fails signature verification and is tombstoned as tampered rather than retried. That root cannot be added after the regions join — see Requirements.

Your applications hold one endpoint throughout and never learn that any of this happened.

Failover: what is automatic and what is not

Catalyst never promotes a database, never demotes one, and never moves traffic between members.

• The traffic half of a failover is automatic. The health check turns and the load balancer follows.
• The database half is not. You promote the replica.

That split is deliberate, for three reasons:

• A promotion cannot be undone. Databases promote a replica so it accepts writes. They do not turn a writer back into a replica. The only way back is to destroy that region's database and rebuild it as a replica of the new one.
• Unreachable is not the same as lost. A health check that has failed for two minutes can mean a gateway restart, an agent restart during a control plane outage, or a network partition between the checker and a region that is serving its clients perfectly well.
• Nothing fences the old writer. An unnecessary promotion leaves two databases accepting writes and diverging, with no way to reconcile them afterwards. The risk of over-eager automation here is not downtime. It is two sources of truth.

You can automate the promotion, and you should if your RTO demands it. Build an automation that waits longer, requires more than one signal, and refuses to promote a replica whose replication lag it cannot read.

Requirements

A group is more than two regions configured alike. The settings below must match in every member. The control plane enforces none of them, and a misconfigured group looks healthy until a failover exposes it.

Setting	Where you set it	What it must be	What happens if it differs
The database	agent.config.project.external_postgresql	One PostgreSQL cluster replicated across the regions, with a single writer. Each region points at the instance next to it.	The members hold unrelated data, and a failover recovers nothing.
Database name and user	Your infrastructure	Identical in every member, character for character.	A promotion becomes a database replacement.
The secrets provider	global.secrets.provider	postgresql in every member, so secrets live in the replicated cluster instead of in one cluster's Kubernetes.	The promoted region cannot read any secret the other one stored. Components fail to initialize, and there is nothing to recover from.
The key encryption key	global.secrets.postgresql.kek_provider	The same key identity, named the same way, at the same key version.	The replicated secret rows are present, but the promoted region cannot decrypt them.
The Dapr scheduler backend	agent.config.internal_dapr.scheduler	The PostgreSQL backend, on the replicated cluster.	Every job and actor reminder is lost at the moment of the failover.
The Dapr PKI root — optional	agent.config.internal_dapr.pki	One root CA, with a per-region intermediate. Needed only if a project in the group signs its workflow history, and it cannot be added after the regions join.	A workflow resumed in another member fails signature verification, and is tombstoned as tampered rather than retried.

The control plane compares two of these and warns without refusing: the key encryption key, and the Dapr trust anchors. Read the group's status after creating it, and before creating any project in it:

diagrid regiongroup get my-group --output json | jq '.status'

An empty .status.messages is the expected result. The status also names which member currently accepts writes.

Deploy a region group on your cloud

A region group is not tied to any one cloud. Catalyst asks the platform underneath for four primitives, and any platform that offers them can host a group.

Platform	Status
AWS	Documented end to end, with Terraform. AWS multi-region deployment builds the four primitives as an RDS cross-region read replica, a KMS multi-region key, and AWS Global Accelerator.
Azure	Supported. No guide yet — build the four primitives yourself, or contact Diagrid. The Azure deployment guide covers the single-region parts.
Other clouds and on-premises Kubernetes	Supported. Any platform providing the four primitives can host a group. Contact Diagrid if you want one reviewed before you build it.

The four primitives:

• A PostgreSQL deployment that replicates across regions, and whose replica can be promoted to accept writes.
• A global load balancer across the member regions that health-checks /diagrid/region/writable and sends traffic to the member answering 200. Prefer one whose address does not change on a failover: the Dapr SDKs hold long-lived gRPC connections that a DNS change would not disturb.
• A key encryption key both regions can use, held outside either cluster.
• One wildcard domain and certificate, served by every member, so the group's projects have one set of hostnames.

Anything else a project depends on that is regional — a pub/sub broker, a rate-limit store — has to be multi-region in its own right. A group fails over Catalyst's own state, not your message broker.

Failover and failback

Every failover is the same two actions, in one order or the other:

• Promote the replica in the member that is taking over. It begins answering 200, and the load balancer starts sending it traffic.
• Rebuild the other member as a replica: destroy its database and rebuild it from the new writer. This is the only demotion available.

Planned failover

Drain the active member first, so writes stop before the database role moves rather than as a consequence of it: drain, promote, let traffic follow, rebuild the drained member as a replica, then undrain it.

Unplanned failover

Skip the drain, because traffic has already left the lost region — its health check stopped passing. But it has not arrived anywhere else: until you promote, the group has no writer and no healthy endpoint.

Promoting has three consequences:

• Whatever had not replicated is lost.
• Writes in flight fail, and are retried by the client.
• A job or reminder can fire a second time, so handlers must be idempotent.

The lost region comes back holding a writer that was never demoted. Keep traffic off it until you have rebuilt it as a replica.

Failback

Failing back is the planned procedure again with the members swapped. Catalyst has no asymmetry here: whichever member holds the writer is the active one.

Recovery objectives (RPO and RTO)

Objective	What determines it	How to measure it
Catalyst publishes no RPO or RTO figure for a region group, because neither is Catalyst's to set. Both follow from your replication lag and your own promotion procedure, so measure them rather than assume them.

Objective	What determines it	How to measure it
RPO — how much data a failover loses	The replication lag at the moment you promote. Cross-region replication is asynchronous, so Catalyst does not bound it.	cra_region_replication_lag_seconds, exported by every passive member. Alert on it, so you know your exposure before a failover rather than after it.
RTO — how long recovery takes	Mostly how long you take to decide to promote. The traffic move is the smaller half: the health check follows the write status within about 20 seconds.	cra_region_writable reports which member accepts writes.

A worked example

A group whose replication lag sits under 5 seconds, operated by a team that takes 10 minutes to confirm the outage and start the promotion:

	Time	Where it goes
RPO	~5 seconds of work	Whatever had not replicated when the region was lost. Workflows resume from their last replicated step and re-run the activities after it.
RTO	~11 minutes	~10 minutes deciding, under a minute promoting, then about 20 seconds for the health check to turn and traffic to follow.

Decision time dominates, which is why the number to improve first is your runbook rather than your infrastructure. Measure your own during the rehearsal below: record the wall-clock time from "we decided to promote" to "the first write succeeded in the new region".

Rehearse a failover on a group that is not serving production before you depend on it. A full round trip rebuilds both members' databases, so plan the rehearsal as maintenance rather than as something you can undo.

Limitations

• The group is not multi-writer. Exactly one member accepts writes at a time, and it is your responsibility to restore that state after a failover has briefly broken it.
• Nothing promotes automatically, and nothing demotes at all. The only demotion is rebuilding a member's database as a replica of the current writer.
• A passive member is a full region. Size every member identically. It runs the same data plane and has to carry the same load.
• You decide membership before the projects exist. A region must be empty to join, a region in a group takes no standalone projects, and neither a project nor a region can change side afterwards.
• Regional services are not failed over. Pub/sub brokers and rate-limit counters are regional. A project in a group brings its own multi-region broker.

Next steps

🌐

AWS multi-region deployment

Build a two-region group on AWS with Terraform.

🔁

AWS failover and failback

The runbook: promote a region, move traffic, and rebuild the old one.

📊

Metrics and alerts

Watch replication lag, and alert when no member — or more than one — accepts writes.

🌍

Regions

Create, inspect, and manage the regions a group is built from.