Infrastructure Connections
Catalyst Connections are exactly what they sound like: a set of resources used to onboard backing infrastructure services into Catalyst so they can be used by the various APIs to perform work.
The majority of supported Catalyst APIs will depend on the existence of at least one Connection to a backing infrastructure service:
- Pub/Sub API: Message broker
- Key/Value API: Key/Value data store
- Workflow API: Workflow data store
- Bindings API: External service to invoke or to receive events from
Catalyst supports creating Connections to many different cloud services in addition to providing managed Connections to Diagrid-hosted services: Diagrid K/V store and Diagrid Pub/Sub.
Overview
To understand the power of Connections, let's consider the scenario of application state management with Redis. Traditionally, a developer would need to import a vendor-specific Redis SDK and author complex plumbing code to configure operations against their cloud-hosted Redis provider of choice. Should the developer ever wish to change the backing state store, an application rewrite is almost inevitable.
Alternatively, the developer could leverage the Catalyst Key/Value API for a code-agnostic, vendor-neutral approach to state management, referencing a Connection and allowing Catalyst to broker the connectivity to the underlying infrastructure resource. The code complexity is significantly reduced, and the developer now has the freedom to switch to another state implementation without changing any application code. In addition, the developer gets end-to-end observability from the application code to the infrastructure layer without any instrumentation, and resiliency policies out-of-the-box for handling transient network failures.
Establishing Infrastructure Connectivity
External Cloud Services
If you have existing cloud services you'd like to use with the Catalyst APIs, you can create an external Connection by providing the following information:
| Settings | Description |
|---|---|
| Type | API related to specified Connection i.e Pub/Sub, Bindings |
| Target | The backing infrastructure resource you want to connect. See a list of supported Connection targets |
| Authentication | The method through which Catalyst will establish a Connection to your target. |
In addition to the above, each connection type will have certain required metadata items which must be provided to establish the infrastructure connection. Optional configuration metadata can also be used to further configure how the target service operates behind the scenes. For a guide on creating external connections, see the Create External Connections guide.
Managed Diagrid Services
To simplify your experience provisioning infrastructure services, Catalyst provides backing services for both the Pub/Sub and Key/Value APIs. These built-in, fully-managed services can be easily deployed on project creation or provisioned ad-hoc with these simple commands: diagrid pubsub create and diagrid kv create.
Like any infrastructure services, Diagrid K/V store and Diagrid Pub/Sub will also have associated Connection resources. These Connections are considered "managed" and have the types pubsub.diagrid and state.diagrid respectively.
Diagrid-managed Connections cannot be deleted directly; the lifecycle of each Connection is scoped to the lifecycle of the Diagrid infrastructure service. To delete a managed Connection, you must delete the Diagrid infrastructure service first.
Limiting Connection Access
By default, a Connection can be used by any App ID- and by extension any connected application- in the same project when making API calls. In order to limit which App IDs can use a given Connection, you can configure Connection scopes.
Access scopes provide a way to restrict which applications can utilize a Connection and the underlying infrastructure it represents. Once scopes are applied, a Connection can only be used in API calls that originate from allowed App IDs.
Configure Resiliency
When creating or updating a Connection, you have the option to enable a set of default resiliency policies. Depending on the API you are using, resiliency policies may be applied to all outbound requests made from an App ID to a backing Connection resource and/or all inbound requests made from your Connection resource to your App ID . In the event of call failures and transient errors, the configured policies would be enforced, without the need for you to write any resiliency code in your application itself.
The same three resiliency policies are available for both outbound and inbound operations, and you can choose which you want to enforce.
Outbound operations are calls from your App ID to the backing Connection resource to accomplish tasks like:
- Persisting or retrieving state
- Publishing a message to a backing Pub/Sub broker
- Invoking an output binding
Inbound operations are calls from your App ID to your external application, such as:
- Pub/Sub subscriptions when delivering a message
- Input bindings
The below policies can be enabled for a given Connection :
Timeout: Early-terminate long-running operations that exceed 10s.
Retry: Retry failed operations
Setting Description policy Constant back-off and retry interval duration 5s between retries maxRetries 5 retries maximum Circuit Breaker: Shut off traffic to the impacted application when certain criteria is met to avoid flooding
Setting Description trip After 5 consecutive failures, the CB will trip and become open timeout 20s after the CB is tripped, it will switch to half-open interval 5s, meaning the CB clears its internal counts after 5s maxRequests 1 request will be allowed to pass through when the CB is half-open and recovering from failure
Only Pub/Sub Connections will make use of both outbound and inbound policies.