Pub/Sub API

Publish-subscribe messaging, also referred to as pub/sub messaging, enables microservices to communicate with each other using messages for event-driven architectures. See Pub/Sub API Reference.

• The producer, or publisher, writes messages to an input channel and sends them to a topic, unaware which application will receive them.
• The consumer, or subscriber, subscribes to the topic and receives messages from an output channel, unaware which service produced these messages.

An intermediary message broker copies each message from a publisher’s input channel to an output channel for all subscribers interested in that message. This pattern is especially useful when you need to decouple microservices from one another.

The Catalyst Pub/Sub API:

• Provides a platform-agnostic API to send and receive messages
• Offers at-least-once message delivery guarantee
• Integrates with various message brokers and queuing systems

The specific message broker used by your application to publish or subscriber to messages is configured using aCatalyst pub/sub component. This removes the infrastructure dependency from your application code, and making it more portable and flexible to changes.

When using Pub/Sub through Catalyst:

• Your application service makes an encrypted request using its App ID Token to the Catalyst Pub/Sub API, targeting a specific Catalyst pub/sub component.
• Catalyst facilitates connectivity to the underlying message broker and publishes the message on behalf of the calling app.
• Catalyst subscribes App IDs to specific pub/sub topics through subscriptions. When messages arrive, they are forwarded to an App ID's configured app endpoint.

Supported Pub/Sub Components

Currently, Catalyst supports 10 components that can be interchangably used with the Pub/Sub API.

Diagrid Pub/Sub

In Catalyst, you can take advantage of Diagrid Pub/Sub, a fully-managed Pub/Sub broker. It is created with a default component, but additional components can be configured.

Catalyst Pub/Sub is only available through the Catalyst APIs.

• AWS SNS/SQS: aws.snssqs
• Azure Event Hubs: azure.eventhubs
• Azure Service Bus Topics: azure.servicebus.topics
• GCP Pub/Sub: gcp.pubsub
• Apache Kafka: kafka
• MQTT3: mqtt3
• Apache Pulsar: pulsar
• RabbitMQ: rabbitmq
• Redis: redis
• Diagrid Pub/Sub: diagrid

Features

Below describe some of the features and characteristics of the Catalyst Pub/Sub API.

Sending messages using Cloud Events

To enable message routing and provide additional context with each message between services, Catalyst uses the CloudEvents 1.0 specification as its message format. Any message sent by an application to a topic using Catalyst is automatically wrapped in a Cloud Events envelope, using Content-Type header value for datacontenttype attribute.

You can optionally disable the CloudEvent wrapping for a publisher or subscriber.

For more information, reference the dapr docs on how to use pub/sub without CloudEvents

Setting message content types

When publishing a message, it's important to specify the content type of the data being sent. Unless specified, Catalyst will assume text/plain.

• HTTP client: the content type can be set in a Content-Type header
• gRPC client and SDK: have a dedicated content type parameter

Message delivery

In principle, the Catalyst-hosted Dapr APIs consider a message successfully delivered once the subscriber processes the message and responds with a non-error response.

Receiving messages with topic subscriptions

Catalyst applications can subscribe to published topics via three subscription types that support the same features: declarative, streaming and programmatic.

Catalyst supports two of these three, declarative and streaming. ❌ Programmatic subscriptions are not supported.

Subscription type	Description
Declarative	The subscription is defined in an external file. The declarative approach removes subscription logic from your code and allows for existing applications to subscribe to topics, without having to change code. To create a declarative subscription, follow the Catalyst documentation.
Streaming	The subscription is defined in the user code. Streaming subscriptions are dynamic, meaning they allow for adding or removing subscriptions at runtime. They do not require an App ID to have an app endpoint, making them easy to configure in code. With streaming subscriptions, since messages are sent to a message handler code, there is no concept of routes or bulk subscriptions. To create a streaming subscription, follow the Dapr documentation to write the application code, and make Catalyst API requests using the Diagrid CLI diagrid dev start --app-id <appid> -- go run <filename>.

Message routing

Catalyst provides content-based routing pattern. Pub/sub routing is an implementation of this pattern that allows developers to use expressions to route CloudEvents based on their contents to different URIs/paths and event handlers in your application.

If no route matches, an optional default route is used. This is useful as your applications expands to support multiple event versions or special cases.

Handling failed messages with dead letter topics

Sometimes, messages can't be processed because of a variety of possible issues, such as erroneous conditions within the producer or consumer application or an unexpected state change that causes an issue with your application code.

In Catalyst, you can set dead letter topics to deal with messages that cannot be delivered to an App ID's application endpoint. This feature is available on all pub/sub components and prevents consumer applications from endlessly retrying a failed message.

Enabling the outbox pattern

Catalyst enables developers to use the Dapr outbox pattern for achieving a single transaction across a transactional state store and any message broker.

Coming soon

A tutorial for using the outbox pattern is in-progress! Stay tuned.

At-least-once guarantee

The Catalyst Pub/Sub API guarantees at-least-once semantics for message delivery. When an application publishes a message to a topic using the pub/sub API, Catalyst ensures the message is delivered at least once to every subscriber.

Even if the message fails to deliver, or your application crashes, Catalyst attempts to redeliver the message until successful delivery.

All Catalyst pub/sub components support the at-least-once guarantee.

Message Time-to-Live (TTL)

Catalyst can set a timeout message on a per-message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This timeout message prevents a build up of unread messages. If a message has been in the queue longer than the configured TTL, it is marked as dead.

Publish and subscribe to bulk messages

Catalyst supports sending and receiving multiple messages in a single request. When writing applications that need to send or receive a large number of messages, using bulk operations allows achieving high throughput by reducing the overall number of requests. For more information, view the bulk Pub/Sub API reference

Try the Pub/Sub API

Walk through the following quickstarts to see Pub/Sub in action:

Quickstart/tutorial	Description
Pub/Sub CLI Quickstart	Try sending and receiving messages using the Diagrid CLI to get started.
Pub/Sub API Quickstart	Send and receive messages using Diagrid Pub/Sub Service