# Observability

Catalyst Enterprise Self-Hosted collects metrics and logs from the Catalyst Dapr Servers. This data is exported to the Diagrid Cloud control plane, where it powers the observability features in the web console.

## OpenTelemetry integration

To forward telemetry to your own observability backend (Grafana Cloud, Mimir, Loki, Tempo, Victoria Metrics, any OTLP-compatible pipeline, etc.), the Catalyst Helm chart bundles two optional [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) deployments, each aligned with one of the standard [collector deployment modes](https://opentelemetry.io/docs/collector/deployment/):

- `opentelemetry-deployment` — [gateway-style](https://opentelemetry.io/docs/collector/deployment/gateway/) `Deployment`-mode collector for traces and service-level metrics.
- `opentelemetry-daemonset` — [agent-style](https://opentelemetry.io/docs/collector/deployment/agent/) `DaemonSet`-mode collector for node-local log collection.

Both are disabled by default. Enable them by setting `enabled: true` and supplying an exporter configuration for your backend. See the [Helm Chart Reference](./reference.md#collector-addons) for the chart values, and the [Collector configuration reference](https://opentelemetry.io/docs/collector/configuration/) for the full set of receivers, processors and exporters.

### Traces and metrics

Enable the deployment-mode collector and configure exporters for your backend. The example below uses the [OTLP HTTP exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter) to forward both traces and metrics to an external endpoint, and scrapes in-cluster metrics via the [Prometheus receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver):

```yaml
opentelemetry-deployment:
  enabled: true
  config:
    exporters:
      otlphttp:
        endpoint: "https://otlp.example.com"
        headers:
          Authorization: "Bearer ${env:OTLP_TOKEN}"
    service:
      pipelines:
        traces:
          receivers: [otlp, zipkin]
          processors: [memory_limiter, batch]
          exporters: [otlphttp]
        metrics:
          receivers: [otlp, prometheus]
          processors: [memory_limiter, batch]
          exporters: [otlphttp]
```

### Logs

Enable the daemonset-mode collector to ship logs from each node's filesystem via the [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) to a Loki or OTLP log endpoint. The example below uses the [Loki exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/lokiexporter) together with the [k8sattributes processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor) to enrich log records with Kubernetes metadata:

```yaml
opentelemetry-daemonset:
  enabled: true
  config:
    exporters:
      loki:
        endpoint: "https://logs.example.com/loki/api/v1/push"
    service:
      pipelines:
        logs:
          receivers: [otlp, filelog]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [loki]
```

### Application tracing

To capture application-level traces, attach a Dapr `Configuration` resource to your app that points at the [OTLP](https://opentelemetry.io/docs/specs/otlp/) endpoint of the deployment-mode collector above (or any other OpenTelemetry-compatible backend). The example below uses a TLS OTLP endpoint and resolves the authentication API key from a Dapr secret store via `secretKeyRef`:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: tracing-config
spec:
  tracing:
    samplingRate: "1"
    otel:
      endpointAddress: "otlp.example.com:4317"
      isSecure: true
      protocol: grpc
      headers:
        - name: "api-key"
          secretKeyRef:
            name: otlp-credentials
            key: api-key
```

Apply it and attach it to an app via the Diagrid CLI:

```bash
diagrid apply -f tracing-config.yaml
diagrid app update <app-id> --app-config tracing-config
```

See the [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/configuration/) and the [Dapr tracing guide](https://docs.dapr.io/operations/observability/tracing/setup-tracing/) for the full set of receivers, processors and exporters.