Metrics and alerts
A region group tells you two things about itself, and both matter before a failover rather than during one:
- How far behind the passive members are. That lag is your RPO — the work a failover would lose. It is worth knowing today, not at the moment you promote.
- Which member accepts writes. Exactly one should. None means the group has no writer and is serving nothing. More than one means two databases are diverging.
Set up both alerts before the group carries production traffic.
Metrics
Every member of a group exports these on the Catalyst agent's /metrics endpoint, alongside the gateway probe metrics that every region exports whether or not it is grouped.
| Metric | Type | Exported by | What it means |
|---|---|---|---|
cra_region_replication_lag_seconds | Gauge, seconds | Passive members | How far this member's database is behind the one accepting writes. Promoting this member now would lose approximately this much work. |
cra_region_writable | Gauge, 1 or 0 | Every member | 1 when this member's database accepts writes. 1 on exactly one member is the healthy state. |
cra_gateway_probe_up | Gauge, 1 or 0 | Every region | Whether this region's own gateway answers. A region with no customer traffic still produces this signal. |
cra_gateway_probe_total | Counter, label result | Every region | Gateway probe outcomes over time. |
A member that reports no cra_region_writable series at all is not reporting 0: its agent has stopped heartbeating, which is a different problem and deserves its own alert.
Scrape them from every member
These metrics only answer the question when you hold them from every member at once. cra_region_replication_lag_seconds is exported by the passive members, so a Prometheus that scrapes only the active region sees nothing, and one that scrapes only its own region cannot tell you how many members accept writes.
Scrape every member into one Prometheus, or federate per-region Prometheus instances into a single view. Whichever you choose, keep a label that identifies the member region, so an alert names which region it is about. See Observability for scraping and OpenTelemetry configuration.
Alerts
The expressions below assume all members are scraped into one Prometheus, and that a region label distinguishes them. Treat the thresholds as starting points and tune them against what you measure during the failover rehearsal.
Replication lag exceeds your RPO. Set the threshold to the data loss you have agreed to accept, so the alert fires while you can still act on it:
- alert: CatalystRegionReplicationLagHigh
expr: cra_region_replication_lag_seconds > 30
for: 5m
annotations:
summary: "Region {{ $labels.region }} is more than 30s behind. A failover now would lose that much work."
No member accepts writes. The group has no writer: either a failover is in progress and nothing has been promoted yet, or the active member's database has stopped accepting writes. Either way the group is serving nothing:
- alert: CatalystRegionGroupNoWriter
expr: max(cra_region_writable) == 0
for: 2m
annotations:
summary: "No member of the region group accepts writes."
Two members accept writes. Both databases are accepting writes and diverging, and there is no way to reconcile them afterwards. This is expected for the few minutes between a promotion and the rebuild that follows it, so page on it only outside that window — or accept the alert during a planned failover and silence it for the duration:
- alert: CatalystRegionGroupTwoWriters
expr: sum(cra_region_writable) > 1
for: 5m
annotations:
summary: "Two members accept writes. Their data is diverging."
A member stopped reporting. Its agent is no longer heartbeating, so nothing above can be trusted for that region:
- alert: CatalystRegionNotReporting
expr: absent_over_time(cra_region_writable[10m])
for: 5m
annotations:
summary: "A region group member has stopped reporting whether it accepts writes."
Check the group without metrics
Two things answer the same question without a monitoring stack, which is useful during an incident and while you are setting monitoring up.
The control plane's view of the group, including which member currently accepts writes and any configuration mismatch it has detected:
diagrid regiongroup get my-group --output json | jq '.status'
Each member's own answer, from its region-specific hostname, which resolves whether or not that region is receiving traffic:
curl -so /dev/null -w '%{http_code}\n' https://catalyst-west.catalyst.example.com/diagrid/region/writable
200 means this member accepts writes, 503 means it runs against a read-only replica, and 404 means it is not in a group. This is the same endpoint the global load balancer health-checks.
