Skip to main content

Apply token budgets to LLM traffic

A token budget caps how many LLM tokens your project can consume in an hour, a day, or a month. In enforce mode Catalyst rejects LLM calls once the allowance is gone; in alert mode it only counts, so you can size an allowance before you commit to it.

For what a budget is and how enforcement behaves, see Token budgets. This page covers creating and running them.

Create a budget in the console​

  1. Open your project in the Catalyst console and go to Monitor › Token Budgets.
  2. Select Create.
  3. Give the budget a name, a token limit, and a window (Hourly, Daily, or Monthly).
  4. Choose a mode: Enforce to reject calls once the allowance is gone, or Alert only to count without rejecting.
  5. Optionally set a scope: a Workload, an LLM component, a Model, or a combination. Leave it empty to cover every LLM call in the project.
  6. Select Create.

The budget appears in the list with its scope, mode, window, current usage, and when the window resets.

Token Budgets list in the Catalyst console showing two hourly budgets on the same scope, one in Enforce mode at 446 of 1k tokens and one in Alert only mode at 444 of 500

The two budgets above are a useful pairing. Both cover the same workload and LLM component, but the alert-mode one has the lower limit, so it runs out first and warns you while the enforce cap still has room. It acts as a tripwire under the ceiling.

Create a budget with the CLI​

Use diagrid tokenbudget create:

# 50M tokens a month across the whole project, enforced
diagrid tokenbudget create monthly-project-cap \
  --limit 50000000 --window month --mode enforce \
  --project my-project

# 1M tokens a day for one agent, counted but never rejected
diagrid tokenbudget create research-agent-daily \
  --limit 1000000 --window day --mode alert \
  --scope-id research-agent \
  --project my-project

Scope selectors are --scope-id (a workload identity), --scope-component (a conversation component), and --scope-model. Set several and they are combined with AND. Omit them all and the budget covers the project.

List and inspect budgets:

diagrid tokenbudget list --project my-project
diagrid tokenbudget get monthly-project-cap -o yaml --project my-project

For every flag, see the Diagrid CLI reference.

Declare a budget in YAML​

TokenBudget is a declarative resource, so budgets can live in Git alongside your other project configuration. See Declarative management for the wider workflow.

# research-agent-daily.yaml
apiVersion: cra.diagrid.io/v1beta1
kind: TokenBudget
metadata:
  name: research-agent-daily
spec:
  limit: 1000000
  window: day        # hour | day | month
  mode: enforce      # enforce | alert
  scope:             # optional; all selectors are AND-combined
    appId: research-agent
    # component: openai-prod
    # model: gpt-4o

Apply it, and re-apply to change it:

diagrid apply -f research-agent-daily.yaml --project my-project

Cover provider SDK calls with the LLM proxy​

Calls made with the Conversation API are already counted. An application that talks to a provider with the provider's own SDK is counted too, once you send it through Catalyst instead of straight to the provider.

Two things change on the client. The base URL points at Catalyst instead of the provider, and a dapr-api-token header carries the workload's API token. Everything else, including your provider key, stays as it is:

import os
from openai import OpenAI

client = OpenAI(
    # The same OpenAI API, reached through Catalyst
    base_url=f"{os.environ['DAPR_HTTP_ENDPOINT']}/v1.0/diagrid/llm/api.openai.com/v1",
    api_key=os.environ["OPENAI_API_KEY"],
    default_headers={"dapr-api-token": os.environ["DAPR_API_TOKEN"]},
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarise this incident report."}],
)

The authentication header​

dapr-api-token is how Catalyst authenticates the call. Without it the request is rejected before it reaches a provider, so this is the one change you cannot skip.

The token is the same one your workload already uses to call any other Catalyst API, published to it as DAPR_API_TOKEN. See Connect to Catalyst for where to find it and your project's HTTP endpoint.

It does more than let the call in. The token identifies which workload is calling, which is what lets a budget scoped to a workload identity recognise the call as yours. Without it Catalyst would have no way to tell one caller's proxied traffic from another's.

Catalyst removes the header once it has checked it, so your Catalyst token is never forwarded to the provider. Your provider key is untouched and the SDK keeps sending it exactly as it would if it were talking to the provider directly.

Other providers and SDKs​

Every provider SDK works the same way, in any language. Keep the client as it is, point its base URL at <your project's HTTP endpoint>/v1.0/diagrid/llm/<provider host>, and add the header. For the Anthropic SDK the base URL is .../v1.0/diagrid/llm/api.anthropic.com, since it appends /v1/messages itself.

Streaming responses work as usual, and are counted when the stream finishes.

Permitted providers

The proxy reaches api.openai.com and api.anthropic.com. A call to any other host is rejected with 403, which stops the proxy being used as general-purpose egress. Ask Diagrid if you need another provider added.

Budgets apply to proxied calls the same way, with one difference: a proxied call involves no Catalyst component, so a budget scoped with --scope-component never matches it. Scope by workload identity or model, or leave the budget project-wide, to cover both routes.

Check what a budget has consumed​

The Usage column in the console shows consumption against the limit, and Resets shows when the window rolls over. The list refreshes on its own, a few seconds behind live traffic.

From the CLI, get returns the same reading under status.utilization:

diagrid tokenbudget get monthly-project-cap -o yaml --project my-project
status:
  utilization:
    consumed: 41283910
    remaining: 8716090
    limit: 50000000
    exhausted: false
    windowEndsAt: "2026-10-01T00:00:00Z"
    observedAt: "2026-09-10T14:32:04Z"
    accuracy: exact

accuracy tells you how much to trust the number. exact means it is the same figure enforcement is working from. partial means Catalyst could only estimate, and the estimate can read higher than what you have really spent, so treat it as a rough guide and don't alert on it.

Get notified when a budget runs out​

You should not have to poll a budget to discover it has been rejecting traffic for an hour. Catalyst raises an ALERT_TYPE_TOKEN_BUDGET_DEPLETED alert when a charge takes a budget to or past its limit, within about half a minute of the crossing. Alerts are listed in the console under Monitor › Alerts:

Alerts page in the Catalyst console showing an ALERT_TYPE_TOKEN_BUDGET_DEPLETED entry with its message and the budget and appid labels

The budget and appid labels name the budget that ran out and the workload that spent the last of the allowance. Routes match on them.

The alert fires once per depletion, not once per rejected request. Everything blocked afterwards is silent, and the window reset re-arms it, so a daily budget that runs dry every afternoon pages you once a day instead of once a call.

Both modes raise it. An alert-mode budget tells you it would have started rejecting traffic, without rejecting any, which is what makes alert mode useful for sizing an allowance.

Deliver it by pairing a destination with a route:

# Where alerts go
diagrid notification destination create ops-webhook \
  --webhook-url https://hooks.example.com/catalyst \
  --project my-project

# Which alerts go there
diagrid notification route create budget-depleted \
  --type ALERT_TYPE_TOKEN_BUDGET_DEPLETED \
  --destination ops-webhook \
  --project my-project

Routes narrow further on the alert's labels, so one team can take only its own agents:

diagrid notification route create research-budget-depleted \
  --type ALERT_TYPE_TOKEN_BUDGET_DEPLETED \
  --filter appid=~research-.* \
  --destination research-oncall \
  --project my-project

A budget whose usage reads accuracy: partial still enforces, but it does not raise this alert.

What callers see when a budget is exhausted​

The call is rejected before it reaches the provider, so it costs nothing. What the caller gets back depends on how it reached Catalyst.

Through the LLM proxy, the rejection comes back as 429 Too Many Requests, which most provider SDKs already understand as a rate limit and back off from.

Through the Conversation API, the rejection arrives as a component failure: HTTP 500 with the error code ERR_CONVERSATION_INVOKE, or gRPC Internal. That is the same status code any conversation failure uses, so the message is what identifies the cause:

failed conversing with component "llm-provider": LLM token budget "monthly-project-cap"
exhausted (50000000 tokens/month); resets at window end

If your application handles budget exhaustion specially, match on that message rather than on the status code, which cannot tell an exhausted budget from an unreachable provider.

An alert-mode budget never rejects, however much it has consumed.

Reset, edit, and delete​

Reset starts the current window's count from zero. The limit, window, mode, and scope are untouched. Use it to restore service after a runaway agent has eaten an allowance you would rather not wait out, or to clear a window you filled up while testing.

In the console, open the budget's row menu and choose Reset, between Edit and Delete. It asks for confirmation first, since resetting an exhausted enforce budget lifts the block for the rest of the window.

From the CLI:

diagrid tokenbudget reset monthly-project-cap --project my-project

That prompts too. Pass --approve to skip the prompt in a script, and --wait to return only once the reset has taken effect:

diagrid tokenbudget reset monthly-project-cap --approve --wait --project my-project

Either way the count restarts within a few seconds rather than instantly. Resetting needs the same permission as editing.

Edit a budget from its row menu in the console, or re-apply its YAML. Changes take effect within seconds, and the usage reading catches up without waiting for more traffic, so you can lower a limit and see straight away where it leaves you.

A new limit governs what you spend from then on and does not rewrite what the window has already used, so setting one below that point exhausts the budget immediately. That does not raise a depletion alert: nothing was spent to cross the line. Changing the window starts the count over. Editing never clears the count on its own. Use Reset for that. A budget's name is fixed: to rename one, create a replacement and delete the original.

Delete removes the budget and everything it has counted:

diagrid tokenbudget delete monthly-project-cap --project my-project

Constraints​

  • A project can hold up to 20 token budgets.
  • Names must start with a lowercase letter and contain only lowercase letters, numbers, and dashes, up to 63 characters. A name is unique within the project.
  • Limits are whole numbers greater than zero, counted as prompt + completion tokens as reported by the provider.

See also​

Last updated on