Alerts and notifications

An alert is a record that a Catalyst component observed something worth notifying you about. Catalyst keeps every alert raised for your organization and shows it in the console. Alerts can also be sent to a webhook by defining a destination and a notification route that points to it.

What raises an alert

Catalyst raises two kinds of alert today:

Alert type	Raised when
ALERT_TYPE_TOKEN_BUDGET_DEPLETED	An LLM token budget consumes its whole allowance for the current window.
ALERT_TYPE_WORKFLOW	A workflow execution reaches an outcome the app asked to be notified about.

Token budget depleted

A depleted budget is reported once per budget window. Requests rejected after the budget runs out do not raise the alert again, and the window resetting re-arms it.

The alert carries four labels:

Label	Value
budget	The token budget's name.
appid	The App ID whose requests consumed the budget.
observed_at	When the depletion was observed, as an RFC 3339 timestamp. It is what makes each window's depletion a distinct alert rather than a repeat of the last one.
window_ends_at	When the allowance returns, as an RFC 3339 timestamp. Omitted when the counter does not report it.

budget and appid are the two worth filtering a route on.

Workflow

An app selects which workflow outcomes are worth an alert — failed, terminated, or stalled — with the --workflow-notifications-* flags on diagrid app create and diagrid app update. Nothing is raised until an app opts in. See Operate workflows for how to configure the policy.

The alert carries five labels a route can filter on:

Label	Value
appid	The App ID the workflow ran under.
workflow_id	The execution's instance id.
workflow_name	The workflow name.
status	failed, terminated or stalled.
parent_workflow_id	The parent execution's instance id. Omitted when the workflow has no parent.

Because the runtime reports an execution's state on every state save, the same outcome can be reported more than once. Repeats carry the same alert and are collapsed rather than delivered again.

Visualizing alerts in the console

The Alerts view lists your alerts newest first, with the time, the alert type, the message, and the labels the alert carries. Opened on a project it shows that project's alerts; opened on the organization it shows every project's, with a project column and a project filter. Both scopes can be filtered by alert type.

Alerts are read-only. There is nothing to acknowledge, resolve, or dismiss.

Delivering alerts to a webhook

Delivery takes two resources, both scoped to a project:

• A notification destination is the sink an alert is delivered to. It is reusable — several routes can name the same destination.
• A notification route selects alerts by their type and their labels, and delivers every match to the destinations it names.

Create a destination

diagrid notification destination create ops-webhook \
  --project my-project \
  --webhook-url https://hooks.example.com/catalyst

To authenticate the delivery, pass a bearer token. The token is moved into the project's secret store on the way in — it is not kept on the destination, and reading the destination back never returns it:

diagrid notification destination create ops-webhook \
  --project my-project \
  --webhook-url https://hooks.example.com/catalyst \
  --webhook-auth-token s3cr3t

Add --webhook-header Name:value for a header the receiver expects, repeating the flag for several.

webhook is the only destination type, and it is the default.

Create a route

A route must specify a name and a destination:

diagrid notification route create all-alerts \
  --project my-project \
  --destination ops-webhook

A route with no filter delivers every alert in the project. Narrow it by specifying an alert type and the labels the alert must match to be delivered to the destination(s). Repeat --filter to define multiple label matches — an alert must satisfy all of them:

diagrid notification route create chat-budget \
  --project my-project \
  --type ALERT_TYPE_TOKEN_BUDGET_DEPLETED \
  --filter budget=chat-tokens \
  --filter appid=~order-.* \
  --destination ops-webhook

Each filter is written as key<op>value:

Operator	Matches when
=	The label equals the value.
!=	The label differs from the value.
=~	The label matches the value as a regular expression.
!~	The label does not match the value as a regular expression.

--type restricts the route to one alert type. Leave it unset to match every type. Repeat --destination to deliver the same matches to several destinations.

Shape the request body

By default the body is the notification serialized as JSON. --webhook-body-template renders it instead, as a Go text template with the notification as the template data: .Type, .Labels, .Message, and .Timestamp.

diagrid notification destination create slack \
  --project my-project \
  --webhook-url https://hooks.slack.com/services/... \
  --webhook-body-template '{"text": {{ .Message | json }}}'

Pipe every interpolated value through the json function, as above, so the body stays valid whatever the value contains.

See also

• diagrid notification reference — every flag on the route and destination commands.
• Observability — metrics, API logs, and topology for the project the alert came from.