Skip to main content

Use Catalyst Quickstart with AWS

In this tutorial, you will select one of the API quickstart applications to try out the Catalyst APIs using AWS resources. The source code used for this tutorial can be found in the catalyst-quickstarts GitHub repository.

Select the API quickstart you would like to deploy to AWS:

pubsub call graph

Prerequisites​

RequirementInstructions
Diagrid AccountTo sign up for Catalyst, visit catalyst.diagrid.io
Diagrid CLIInstall the Diagrid CLI
GitHub AccountGet one for free
AWS accountFor getting started on AWS, visit create and activate an AWS account
AWS CLIInstall the AWS CLI

Log in to AWS​

To authenticate to AWS from the AWS CLI, use short-term access credentials. You can set these credentials to automatically refresh upon expiration, or you can manually configure the temp credentials using environment variables to get up and running quickly.

As an example, you can set your AWS access credentials and preferred deployment region as environment variables which are then used to configure AWS access from the CLI.

export AWS_ACCESS_KEY_ID=XXX
export AWS_SECRET_ACCESS_KEY=YYY
export AWS_DEFAULT_REGION=us-east-1

aws configure
Tip

Need more support configuring access to AWS from the AWS CLI?

  • Follow the instructions here for assistance generating short-term AWS access credentials.
  • See the AWS docs on setting up the AWS CLI.

Deploy infrastructure services to AWS​

Select the API quickstart you plan to deploy to view the relevant instructions.

This tutorial will use SNS/SQS for the Pub/Sub component target. SNS topics and SQS subscriptions are provisioned on-demand by the Catalyst SNS/SQS component, therefore no manual deployment steps are required.

Prepare Diagrid Catalyst resources​

Log in​

Authenticate to your Diagrid Catalyst organization using the following command:

diagrid login

Confirm your login was successful:

diagrid whoami

Create project​

If you do not have an existing project available within your organization, create a new Catalyst project.

diagrid project create catalyst-project -w

To ensure this project is set as the default project in the Diagrid CLI, run:

diagrid project use catalyst-project

Retrieve the HTTP and gRPC endpoints for your Catalyst project. These endpoints will be used to make requests to Catalyst from the quickstart application.

export DAPR_HTTP_ENDPOINT=$(diagrid project get -o json | jq -r '.status.endpoints.http.url')
export DAPR_GRPC_ENDPOINT=$(diagrid project get -o json | jq -r '.status.endpoints.grpc.url')
Confirm value set
echo $DAPR_HTTP_ENDPOINT && echo $DAPR_GRPC_ENDPOINT

Create App IDs​

In Catalyst, each application is represented via a corresponding remote identity, known as an App ID. An App ID functions as the single point of contact for all interactions between a specific application and the Catalyst APIs.

Create two App IDs: one for the publishing app and one for the subscribing app.

diagrid appid create publisher && diagrid appid create subscriber

cli appid output

Retrieve the API token for the publisher App ID. App ID API tokens are used to authenticate API requests to Catalyst.

export PUBLISHER_API_TOKEN=$(diagrid appid get publisher -o json  | jq -r '.status.apiToken')
Confirm value set
echo $PUBLISHER_API_TOKEN

Onboard AWS infrastructure to Catalyst​

In order to use SNS/SQS with the Pub/Sub API, you need represent that resource in Catalyst via a Pub/Sub Component. The component contains the authentication information needed by Catalyst to securely access the AWS resource.

Create AWS access keys​

In order to authenticate to AWS infrastructure from a component, an identity is required which has the necessary permissions to interact with the target resource- AWS SNS/SQS. The following permissions will be attached to the IAM user:

  • AmazonSNSFullAccess
  • AmazonSQSFullAccess

Create a new IAM user:

aws iam create-user --user-name catalyst-pubsub

Attach the required SNS/SQS policies:

# Attach the AmazonSNSFullAccess policy
aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AmazonSNSFullAccess --user-name catalyst-pubsub
# Attach the AmazonSQSFullAccess policy
aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AmazonSQSFullAccess --user-name catalyst-pubsub

Create an access key:

aws iam create-access-key --user-name catalyst-pubsub

Copy the AccessKeyId and the SecretAccessKey from the response:

{
    "AccessKey": {
        "UserName": "catalyst-pubsub",
        "AccessKeyId": <your-access-key>,
        "Status": "Active",
        "SecretAccessKey": <your-secret-key>,
        "CreateDate": "2024-05-08T02:33:38+00:00"
    }
}

Store the output as environment variables:

export COMPONENT_ACCESS_KEY=<your-access-key>
export COMPONENT_SECRET_KEY=<your-secret-key>

Create pub/sub component​

Now that we have retrieved the appropriate authentication credentials, we can create the AWS infrastructure components. There are several ways to create components as described in Connect to external infrastructure. In this tutorial, we will use the non-prompt based CLI experience.

Create the AWS SNS/SQS Pub/Sub Component:

 diagrid component create aws-pubsub --type pubsub.aws.snssqs --metadata accessKey=$COMPONENT_ACCESS_KEY --metadata secretKey=$COMPONENT_SECRET_KEY --metadata awsRegion=$AWS_DEFAULT_REGION --scopes publisher,subscriber
Check resource creation
diagrid component get aws-pubsub

Create topic subscription​

To configure the subscriber app to receive messages published by the publisher app through the Pub/Sub API, create a topic subscription targeting the aws-pubsub component with the following details:

  • Subscription name: aws-subscription
  • Target Pub/Sub component: aws-pubsub
  • Topic on which to listen for messages: orders
  • Default route on the subscriber app for message delivery: /neworder
  • Subscribing App IDs (scopes): subscriber
diagrid subscription create aws-subscription --component aws-pubsub --topic orders --route /neworder --scopes subscriber
Check resource creation
diagrid subscription get aws-subscription

Deploy containerized quickstart applications to AWS App Runner​

This tutorial uses the quickstart apps available in the catalyst-quickstarts GitHub repository. These applications have been containerized and pushed to a public registry.

For the Pub/Sub quickstart, you will deploy two services:

  • publisher app which makes requests to the Catalyst Publish API
  • subscriber app which subscribes to and receives the messages published by the publisher app through Catalyst

To make API requests to Catalyst, the publisher application uses the OSS Dapr SDKs. Environment variables will be configured to enable the Dapr Client to connect to Catalyst instead of a locally-running Dapr sidecar.

Learn more about using the Dapr SDKs to access Catalyst APIs here.

Run the below command to specify a language for the quickstart:

export LANGUAGE=python

Deploy publisher application​

Deploy the publisher application to App Runner. This command will:

  • Deploy the sample application in the language of your choice to App Runner
  • Configure the environment variables required for the application to invoke Catalyst API calls
  • Expose the App Runner instance on a publically-accessible endpoint
aws apprunner create-service --service-name publisher \
--source-configuration '{
    "ImageRepository": {
        "ImageIdentifier": "public.ecr.aws/diagrid/pubsub-'${LANGUAGE}'-publisher-qs:latest",
        "ImageRepositoryType": "ECR_PUBLIC",  
        "ImageConfiguration": { "RuntimeEnvironmentVariables": {
            "DAPR_HTTP_ENDPOINT": "'${DAPR_HTTP_ENDPOINT}'",
            "DAPR_API_TOKEN" : "'${PUBLISHER_API_TOKEN}'",
            "DAPR_GRPC_ENDPOINT" : "'${DAPR_GRPC_ENDPOINT}'",
            "PUBSUB_NAME": "aws-pubsub"
        }, "Port": "5002" }},
    "AutoDeploymentsEnabled": false }' \
--instance-configuration '{ "Cpu": "512", "Memory": "1024"}' \
--network-configuration '{"EgressConfiguration": { "EgressType": "DEFAULT" }, "IngressConfiguration": {"IsPubliclyAccessible": true}}'
Important

It may take up to 5 minutes for the App Runner deployment to successfully complete.

Deploy subscriber application​

Deploy the subscriber application to App Runner. This command will:

  • Deploy the sample application in the language of your choice to App Runner
  • Expose the App Runner instance on a publically-accessible endpoint
aws apprunner create-service --service-name subscriber \
--source-configuration '{
    "ImageRepository": {
        "ImageIdentifier": "public.ecr.aws/diagrid/pubsub-'${LANGUAGE}'-subscriber-qs:latest",
        "ImageRepositoryType": "ECR_PUBLIC",  
        "ImageConfiguration": { "Port": "5002" }},
    "AutoDeploymentsEnabled": false }' \
--instance-configuration '{ "Cpu": "512", "Memory": "1024"}' \
--network-configuration '{"EgressConfiguration": { "EgressType": "DEFAULT" }, "IngressConfiguration": {"IsPubliclyAccessible": true}}'
Important

It may take up to 5 minutes for the App Runner deployment to successfully complete.

Update the subscriber App ID with an app endpoint​

Retrieve the FQDN for the subscriber application and associate it with the subscriber App ID to ensure incoming requests and events from the Catalyst APIs are sent to the running application.

export SUBSCRIBER_APP_ENDPOINT=$(aws apprunner list-services | jq -r '.ServiceSummaryList[] | select(.ServiceName=="subscriber") | .ServiceUrl')
Confirm value set
echo $SUBSCRIBER_APP_ENDPOINT

To ensure incoming App ID requests and events can be forwarded to the subscriber application, update the subscriber App ID with the subscriber application endpoint and wait for the operation to complete.

diagrid appid update subscriber --app-endpoint https://$SUBSCRIBER_APP_ENDPOINT -w

Interact with the Catalyst APIs​

You've now successfully deployed the application code and all of the necessary Catalyst resources to begin invoking the Pub/Sub API.

Retrieve the App Runner URL of the publisher app which will be used to send requests.

export PUBLISHER_APP_ENDPOINT=$(aws apprunner list-services | jq -r '.ServiceSummaryList[] | select(.ServiceName=="publisher") | .ServiceUrl')
Confirm value set
echo $PUBLISHER_APP_ENDPOINT

Use this command to kick off API calls from the publisher application which in turn uses the Catalyst Publish API.

curl -i -X POST https://$PUBLISHER_APP_ENDPOINT/order  -H "Content-Type: application/json" -d '{"orderId":1}'

Check API Logs​

Navigate to the Catalyst console to confirm the request was successful using the API Logs. You should see two requests:

  • An inbound request from the publisher app using the publisher App ID API token to publish the order message
  • An outbound request from the subscriber App ID to deliver the message to the configured AWS app endpoint

pubsub api logs subscriber api logs

Clean up resources​

You can use these commands to clean up the AWS and Catalyst resources you created once you are done using them.

Delete App Runner instances​

aws apprunner delete-service --service-arn=`aws apprunner list-services | jq -r '.ServiceSummaryList[] | select(.ServiceName=="publisher") | .ServiceArn'`
aws apprunner delete-service --service-arn=`aws apprunner list-services | jq -r '.ServiceSummaryList[] | select(.ServiceName=="subscriber") | .ServiceArn'`

Delete SQS queue, SNS topic, and subscription​

export TOPIC_ARN=$(aws sns list-topics --query "Topics[?ends_with(TopicArn, 'orders')].TopicArn | [0]" --output text)
export SUBSCRIPTION_ARN=$(aws sns list-subscriptions-by-topic --topic-arn=$TOPIC_ARN | jq -r '.Subscriptions[0].SubscriptionArn')
export QUEUE_URL=$(aws sqs list-queues --queue-name-prefix=subscriber | jq -r '.QueueUrls[0]')

aws sns unsubscribe --subscription-arn=$SUBSCRIPTION_ARN
aws sns delete-topic --topic-arn=$TOPIC_ARN
aws sqs delete-queue --queue-url=$QUEUE_URL

Delete Catalyst resources​

diagrid appid delete publisher 
diagrid appid delete subscriber
diagrid component delete aws-pubsub

If you want to delete the entire Catalyst project, including the managed infrastructure resources, run the diagrid project delete command.

diagrid project delete catalyst-project