Notifications
Notifications in Conductor help proactively detect potential issues with a cluster's Dapr installation and Dapr-enabled apps by continuously evaluating signals of interest from the monitoring and logging system.
By default, Conductor provides a set of notification rules, which when triggered, will propogate events to the Conductor event stream via the two default channels for metrics and logs. In Conductor Enterprise, custom notifications can be configured and delivered by creating the following:
- Rules: Define the criteria and conditions for triggering events of specific severities.
- Channels: Act as subscribers connecting the events generated by a rule with an event consumption channel such as the Conductor UI, email, or webhook.
Notification rules
A notification rule will continuously monitor Dapr-enabled app and sidecar telemetry and Dapr log signals to evaluate whether to fire a notification with a specific severity.
Rules are composed of the following:
- Alert type: The category of data to act on (
metrics,logsoradvisor) - Target type: The target resource to monitor (
cluster,application,component) - Severity: The severity of the event triggered by the rule (
criticalorwarning) - Name: A descriptive name that helps identify the rule
- Conditions: The conditions which will trigger the rule to fire an event based on the alert type selected
When you create a new organization in Conductor, you get a default set of notification rules and channels set up out of the box. These notication rules are created to ensure critical metrics issues, high advisories, and log warnings are easily surfaced.
Default rules
- Enterprise
- Free
| Name | Type | Severity | Target | Conditions |
|---|---|---|---|---|
| Default App Crash | metrics | critical | all apps | App container in CrashLoopBackOff state for > 2 minutes |
| Default Dapr Crash | metrics | critical | all apps | Dapr sidecar (Daprd) in CrashLoopBackOff state for > 2 minutes |
| Default App to Dapr HTTP Error Rate | metrics | critical | all apps | > 30% for > 2m |
| Default App to Dapr gRPC Error Rate | metrics | critical | all apps | > 30% for > 2m |
| Default Dapr to App HTTP Error Rate | metrics | critical | all apps | > 30% for > 2m |
| Default Dapr to App gRPC Error Rate | metrics | critical | all apps | > 30% for > 2m |
| Default Component Error Rate | metrics | critical | all components | > 30% for > 2m |
| Default Dapr Operator Unavailable | metrics | critical | all clusters | > 30% for > 2m |
| Default Dapr Sentry Unavailable | metrics | critical | all clusters | 0 sentry pods in Running status > 2m |
| Default Dapr Placement Unavailable | metrics | critical | all clusters | 0 placement pods Running status > 2m |
| Default Dapr Sidecar Injector Unavailable | metrics | critical | all clusters | 0 injector pods Running status > 2m |
| Default Dapr Sentry Root Cert Expiry | metrics | critical | all clusters | Dapr Cert detected to expire in < 10d |
| Default High Impact Advisories | metrics | critical | all clusters | Any advisories with severity high present for > 15m |
| Default Logs | logs | warning | all clusters | Any Daprd logs with level fatal, error, or warning |
| Name | Type | Severity | Target | Conditions |
|---|---|---|---|---|
| Default App Crash | metrics | critical | all apps | App container in CrashLoopBackOff state for > 2 minutes |
| Default Dapr Crash | metrics | critical | all apps | Dapr sidecar (Daprd) in CrashLoopBackOff state for > 2 minutes |
| Default Dapr Operator Unavailable | metrics | critical | all clusters | > 30% for > 2m |
| Default Dapr Sentry Unavailable | metrics | critical | all clusters | 0 sentry pods running > 2m |
| Default Dapr Placement Unavailable | metrics | critical | all clusters | 0 placement pods running > 2m |
| Default Dapr Sidecar Injector Unavailable | metrics | critical | all clusters | 0 injector pods running > 2m |
| Default Dapr Sentry Root Cert Expiry | metrics | critical | all clusters | Dapr Cert detected to expire in < 10d |
| Default App to Dapr HTTP Error Rate | metrics | warning | all apps | > 0% for > 2m |
| Default App to Dapr gRPC Error Rate | metrics | warning | all apps | > 0% for > 2m |
| Default Dapr to App HTTP Error Rate | metrics | warning | all apps | > 0% for > 2m |
| Default Dapr to App gRPC Error Rate | metrics | warning | all apps | > 0% for > 2m |
| Default Component Error Rate | metrics | warning | all components | > 0% for > 2m |
| Default Logs | logs | warning | all clusters | Any Daprd logs with level fatal, error, or warning |
Custom Rules
- Enterprise
- Free
Conductor Enterprise supports creating custom rules using the metrics below:
| Target | Name | Description |
|---|---|---|
| cluster | Dapr Control Plane CPU Usage | The CPU usage of the Dapr control plane services |
| cluster | Dapr Control Plane Memory Usage | The memory usage of the Dapr control plane services |
| cluster | Dapr Sentry Root Cert Expiry | The remaining time until the Dapr sentry root certificate expires |
| application | App to Dapr HTTP Latency 95th Percentile | The 95th percentile latency of HTTP requests to the Dapr API |
| application | App to Dapr HTTP RPS | The number of HTTP requests per second to the Dapr API |
| application | App to Dapr HTTP Error Rate | The error rate of HTTP requests to the Dapr API |
| application | Dapr to App HTTP Latency 95th Percentile | The 95th percentile latency of HTTP requests from Dapr |
| application | Dapr to App HTTP RPS | The number of HTTP requests per second from Dapr |
| application | Dapr to App HTTP Error Rate | The error rate of HTTP requests from Dapr |
| application | App to Dapr gRPC Latency 95th Percentile | The 95th percentile latency of gRPC requests to the Dapr API |
| application | App to Dapr gRPC RPS | The number of gRPC requests per second to the Dapr API |
| application | App to Dapr gRPC Error Rate | The error rate of gRPC requests to the Dapr API |
| application | Dapr to App gRPC Latency 95th Percentile | The 95th percentile latency of gRPC requests from Dapr |
| application | Dapr to App gRPC RPS | The number of gRPC requests per second from Dapr |
| application | Dapr to App gRPC Error Rate | The error rate of gRPC requests from Dapr |
| application | App CPU Usage | The CPU usage of the app |
| application | App Memory Usage | The memory usage of the app |
| application | Dapr CPU Usage | The CPU usage of the Dapr sidecar |
| application | Dapr Memory Usage | The memory usage of the Dapr sidecar |
| application | App Container Restart Count | The restart count of the app container |
| application | Dapr Sidecar Container Restart Count | The restart count of the Dapr sidecar container |
| application | App Container Restart Count | The restart count of the app container |
| application | Dapr Sidecar Container Crash Status | The crash status of the Dapr sidecar container |
| application | App Container Crash Status | The crash status of the app container |
| component | App Component RPS | The number of requests per second made to components by the app |
| component | App Component Latency 95 Percentile | The 95th percentile latency of requests made to components by the app |
| component | App Component Error Rate | The error rate of requests made to components by an app |
Custom rules are not supported in Conductor Free.
Notification channels
Notification channels are the subscribers that receive notifications fired by the rules. All notifications with a matching target and severity will be forwarded to the destination specified in the channel.
Channels are composed of the following:
- Channel ID: An auto-generated, unique identifier for a specific channel
- Alert type: The category of nofication events to deliver (
metrics,logsoradvisor) - Severities: The severity level(s) which should be delivered upon firing
- Target clusters: The clusters in which this notification will be active
- Destination: The delivery target where the matching events will be sent
Default channels
By default, two notification channels are created in an organization:
- A channel to send
metricsevents of severitywarningandcriticalto the Conductor events dashboard. - A channel to send
logsevents of severitywarningandcriticalto the Conductor events dashboard.
Conductor Free only provides support for the default channels. To create custom notification channels, use Conductor Enterprise.
Notification events
Conductor provides a dashboard that displays all notifications sent to the Conductor channel across all connected clusters. The dashboard allows filtering on:
- Type - logs, metrics, or advisor event type.
- Log Level - filter based on log levels
error,warningandfatal. Filter only applies to log event types. - Time Span - filter events that occur in the last X hours or X days.
- Metric Severity - filter based on severity values
criticalandwarning. Filter only applies to metrics and advisor event types. - Cluster - events from a specific cluster.
- Namespace - events from a specific cluster namespace.
- Dapr Application Id - events from a specific application specified by its Dapr App ID.
- Message - performs a fuzzy text-based search of the
Messagefield of events.
Clicking on a notification entry brings up the details panel for viewing the full notification details and navigating to the source cluster, application or component that generated the event.
Conductor automatically deduplicates repeat log entries for the same app so that they only show up hourly in the Notifications Events dashboard. Thus, if there is a log entry that is constantly output to the Dapr sidecar it will only show up once an hour in Conductor.