# Metrics and alerts

A [region group](https://docs.diagrid.io/operate/platform-operations/multi-region) reports two things you want to know before a failover, not during one:

- **How far behind the passive members are.** This lag is your [RPO](https://docs.diagrid.io/operate/platform-operations/multi-region#recovery-objectives-rpo-and-rto): the work a failover would lose right now.
- **Which member accepts writes.** It should be exactly one. None means the group has no writer and serves nothing. More than one means two databases are drifting apart.

Set up alerts for both before the group handles production traffic.

## Metrics

Every member exports these on the Catalyst agent's `/metrics` endpoint. They sit next to the [gateway probe metrics](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/reference#gateway-probe), which every region exports, whether it's in a group or not.

| Metric | Type | Exported by | What it means |
|---|---|---|---|
| `cra_region_replication_lag_seconds` | Gauge, seconds | Passive members | How far this member's database is behind the one that accepts writes. If you promoted this member now, you'd lose about this much work. |
| `cra_region_writable` | Gauge, `1` or `0` | Every member | `1` when this member's database accepts writes. In a healthy group, exactly one member reports `1`. |
| `cra_gateway_probe_up` | Gauge, `1` or `0` | Every region | Whether this region's gateway responds. You get this signal even when the region has no customer traffic. |
| `cra_gateway_probe_total` | Counter, label `result` | Every region | Gateway probe results over time. |

If a member reports no `cra_region_writable` at all, that's not the same as `0`. It means the member's agent has stopped sending heartbeats, which is a separate problem that needs its own alert.

## Scrape them from every member

You need these metrics from every member at once. Only passive members export `cra_region_replication_lag_seconds`, so a Prometheus that scrapes only the active region never sees it. And a Prometheus that scrapes only its own region can't tell you how many members accept writes.

Scrape every member into one Prometheus, or federate one Prometheus per region into a single view. Either way, keep a label that identifies each member region, so your alerts say which region they're about. See [Observability](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/observability) to set up scraping and OpenTelemetry.

To check a member's values by hand, port-forward to the agent. You can't exec into it, because the agent image is distroless and has no shell.

```bash
kubectl -n cra-agent port-forward deploy/agent 9090:9090
curl -s localhost:9090/metrics | grep cra_region_
```

## Alerts

These rules assume every member is scraped into one Prometheus, with a `region` label to tell them apart. The thresholds are starting points. Adjust them based on what you measure when you [rehearse a failover](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-deployment#11-rehearse-a-failover).

**Replication lag is higher than your RPO.** Set the threshold to the amount of data loss you've agreed to accept, so the alert fires while you can still do something about it:

```yaml
- alert: CatalystRegionReplicationLagHigh
  expr: cra_region_replication_lag_seconds > 30
  for: 5m
  annotations:
    summary: "Region {{ $labels.region }} is more than 30s behind. A failover now would lose that much work."
```

**No member accepts writes.** Either a failover is underway and you haven't promoted a replica yet, or the active member's database stopped accepting writes. Either way, the group is serving nothing:

```yaml
- alert: CatalystRegionGroupNoWriter
  expr: max(cra_region_writable) == 0
  for: 2m
  annotations:
    summary: "No member of the region group accepts writes."
```

**Two members accept writes.** Both databases are taking writes and drifting apart, and you can't merge them later. This is expected for a few minutes during a planned failover, between promoting one member and rebuilding the other. Silence the alert for that window, and page on it any other time:

```yaml
- alert: CatalystRegionGroupTwoWriters
  expr: sum(cra_region_writable) > 1
  for: 5m
  annotations:
    summary: "Two members accept writes. Their data is diverging."
```

**A member stopped reporting.** Its agent has stopped sending heartbeats, so you can't trust the other alerts for that region:

```yaml
- alert: CatalystRegionNotReporting
  expr: absent_over_time(cra_region_writable[10m])
  for: 5m
  annotations:
    summary: "A region group member has stopped reporting whether it accepts writes."
```

## Check the group without metrics

You can answer the same questions without a monitoring stack. This helps during an incident, or before your monitoring is set up.

Ask the control plane. Its view of the group includes which member accepts writes and any mismatched settings it has found:

```bash
diagrid region group get my-group --output json | jq '.status'
```

Or ask each member directly, using its region-specific hostname. That hostname works whether or not the region is receiving traffic:

```bash
curl -so /dev/null -w '%{http_code}\n' https://catalyst-west.catalyst.example.com/diagrid/region/writable
```

`200` means the member accepts writes, `503` means it runs against a read-only replica, and `404` means it isn't in a group. The global load balancer health-checks this same endpoint.

## Next steps

- [Multi-region high availability](https://docs.diagrid.io/operate/platform-operations/multi-region) — What a region group is, what its members share, and how a failover works.
- [AWS failover and failback](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/aws-multi-region-failover) — The runbook to follow when these alerts fire, on AWS.
- [Azure failover and failback](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/azure-multi-region-failover) — The same runbook on Azure, where promotion works differently.
- [Observability](https://docs.diagrid.io/operate/hosting/enterprise-self-hosted/observability) — Scrape metrics and export traces and logs from a self-managed region.