# Alerts and notifications

An **alert** is a record that a Catalyst component observed something worth notifying you about. Catalyst keeps every alert raised for your organization and shows it in the console. Alerts can also be sent to a webhook by defining a **destination** and a **notification route** that points to it.

## What raises an alert

Catalyst raises two kinds of alert today:

| Alert type | Raised when |
| --- | --- |
| `ALERT_TYPE_TOKEN_BUDGET_DEPLETED` | An LLM token budget consumes its whole allowance for the current window. |
| `ALERT_TYPE_WORKFLOW` | A workflow execution reaches an outcome the app asked to be notified about. |

### Token budget depleted

A depleted budget is reported once per budget window. Requests rejected after the budget runs out do not raise the alert again, and the window resetting re-arms it.

The alert carries four labels:

| Label | Value |
| --- | --- |
| `budget` | The token budget's name. |
| `appid` | The App ID whose requests consumed the budget. |
| `observed_at` | When the depletion was observed, as an RFC 3339 timestamp. It is what makes each window's depletion a distinct alert rather than a repeat of the last one. |
| `window_ends_at` | When the allowance returns, as an RFC 3339 timestamp. Omitted when the counter does not report it. |

`budget` and `appid` are the two worth filtering a route on.

### Workflow

An app selects which workflow outcomes are worth an alert — failed, terminated, or stalled — with the `--workflow-notifications-*` flags on `diagrid app create` and `diagrid app update`. Nothing is raised until an app opts in. See [Operate workflows](https://docs.diagrid.io/operate/project-operations/workflows) for how to configure the policy.

The alert carries five labels a route can filter on:

| Label | Value |
| --- | --- |
| `appid` | The App ID the workflow ran under. |
| `workflow_id` | The execution's instance id. |
| `workflow_name` | The workflow name. |
| `status` | `failed`, `terminated` or `stalled`. |
| `parent_workflow_id` | The parent execution's instance id. Omitted when the workflow has no parent. |

Because the runtime reports an execution's state on every state save, the same outcome can be reported more than once. Repeats carry the same alert and are collapsed rather than delivered again.

## Visualizing alerts in the console

The **Alerts** view lists your alerts newest first, with the time, the alert type, the message, and the labels the alert carries. Opened on a project it shows that project's alerts; opened on the organization it shows every project's, with a project column and a project filter. Both scopes can be filtered by alert type.

Alerts are read-only. There is nothing to acknowledge, resolve, or dismiss.

## Delivering alerts to a webhook

Delivery takes two resources, both scoped to a project:

- A **notification destination** is the sink an alert is delivered to. It is reusable — several routes can name the same destination.
- A **notification route** selects alerts by their type and their labels, and delivers every match to the destinations it names.

### Create a destination

```bash
diagrid notification destination create ops-webhook \
  --project my-project \
  --webhook-url https://hooks.example.com/catalyst
```

To authenticate the delivery, pass a bearer token. The token is moved into the project's secret store on the way in — it is not kept on the destination, and reading the destination back never returns it:

```bash
diagrid notification destination create ops-webhook \
  --project my-project \
  --webhook-url https://hooks.example.com/catalyst \
  --webhook-auth-token s3cr3t
```

Add `--webhook-header Name:value` for a header the receiver expects, repeating the flag for several.

`webhook` is the only destination type, and it is the default.

### Create a route

A route must specify a name and a destination:

```bash
diagrid notification route create all-alerts \
  --project my-project \
  --destination ops-webhook
```

A route with no filter delivers every alert in the project. Narrow it by specifying an alert type and the labels the alert must match to be delivered to the destination(s). Repeat `--filter` to define multiple label matches — an alert must satisfy all of them:

```bash
diagrid notification route create chat-budget \
  --project my-project \
  --type ALERT_TYPE_TOKEN_BUDGET_DEPLETED \
  --filter budget=chat-tokens \
  --filter appid=~order-.* \
  --destination ops-webhook
```

Each filter is written as `key<op>value`:

| Operator | Matches when |
| --- | --- |
| `=` | The label equals the value. |
| `!=` | The label differs from the value. |
| `=~` | The label matches the value as a regular expression. |
| `!~` | The label does not match the value as a regular expression. |

`--type` restricts the route to one alert type. Leave it unset to match every type. Repeat `--destination` to deliver the same matches to several destinations.

### Shape the request body

By default the body is the notification serialized as JSON. `--webhook-body-template` renders it instead, as a Go text template with the notification as the template data: `.Type`, `.Labels`, `.Message`, and `.Timestamp`.

```bash
diagrid notification destination create slack \
  --project my-project \
  --webhook-url https://hooks.slack.com/services/... \
  --webhook-body-template '{"text": {{ .Message | json }}}'
```

Pipe every interpolated value through the `json` function, as above, so the body stays valid whatever the value contains.

## See also

- [`diagrid notification` reference](https://docs.diagrid.io/references/catalyst/cli-reference/notification) — every flag on the route and destination commands.
- [Observability](https://docs.diagrid.io/operate/project-operations/observability) — metrics, API logs, and topology for the project the alert came from.