Using Catalyst APIs with AWS Resources

Deploy two generic applications to AWS to interact with the Pub/Sub, Request/Reply and Key/Value APIs. In this tutorial, you will:

1. Deploy and connect to AWS infrastructure services to configure the Catalyst APIs: DynamoDB and AWS SNS/SQS
2. Deploy two sample applications to AWS App Runner: An inbound application to test calling Catalyst APIs, and an outbound application to receive the resulting calls/events.
3. Create and configure Diagrid App IDs to connect to and from the AWS applications
4. Invoke the Catalyst Pub/Sub, Request/Reply and Key/Value APIs

The source code used for this tutorial can be found in the catalyst-samples GitHub repository.

Prerequisites

Requirement	Instructions
Diagrid Account	Catalyst is currently an early-access product. If you have not already been invited to create an account, join the waitlist to request access.
Diagrid CLI	Install the Diagrid CLI
GitHub Account	Get one for free.
AWS account	For getting started on AWS, visit create and activate an AWS account.
AWS CLI	Install the AWS CLI.

Log in to AWS

To authenticate to AWS from the AWS CLI, it is recommended to use short-term access credentials. You can set these credentials to automatically refresh upon expiration, or you can manually configure the temp credentials using environment variables to get up and running quickly.

As an example, you can set your AWS access credentials and preferred deployment region as environment variables which are then used to configure AWS access from the CLI.

export AWS_ACCESS_KEY_ID=XXX
export AWS_SECRET_ACCESS_KEY=YYY
export AWS_DEFAULT_REGION=us-east-1

aws configure

Need more support configuring access to AWS from the AWS CLI?

• Follow the instructions here for assistance generating short-term AWS access credentials.
• See the AWS docs on setting up the AWS CLI.

Deploy infrastructure services to AWS

This tutorial will require deploying the following AWS infrastructure resources:

• SNS/SQS for pub/sub: SNS topics and SQS subscriptions will be provisioned on-demand by the Catalyst connection, therefore no manual deployment steps are required.
• DynamoDB for key/value state management: A DynamoDB table will be created for persisting key/value pairs.

Deploy DynamoDB KV store

Create an AWS DynamoDB table which will be used as the backing store for the Key/Value API.

aws dynamodb create-table --table-name orders --attribute-definitions AttributeName=key,AttributeType=S --key-schema AttributeName=key,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 

Create Diagrid Catalyst resources

Log in

Authenticate to your Diagrid Catalyst organization using the following command:

diagrid login 

Confirm your login was successful:

diagrid whoami

Create project

If you do not have an existing project available within your organization, create a new Catalyst project.

diagrid project create catalyst-project

To ensure this project is set as the default project in the Diagrid CLI, run:

diagrid project use catalyst-project

Create App IDs

In Catalyst, each application is represented via a corresponding remote identity, known as an App ID. An App ID functions as the single point of contact for all interactions between a specific application and the Catalyst APIs.

Create two App IDs: One for the sample app which will make inbound calls to Catalyst, and the other for the sample app which will receive outbound calls from Catalyst.

diagrid appid create inbound 
diagrid appid create outbound

Retrieve the API token for the inbound App ID. App ID API tokens are used to authenticate API requests to Catalyst from external callers.

export INBOUND_API_TOKEN=$(diagrid appid get inbound -o json  | jq -r '.status.apiToken')

Recommended: Confirm successful retrieval by running echo $INBOUND_API_TOKEN

Create AWS infrastructure connections

In order to use the infrastructure resources deployed above with the Catalyst APIs, two connections are needed:

• aws-pubsub: Pub/Sub Connection for the SNS/SQS broker
• aws-kvstore: KV Connection for the DynamoDB state store

Create AWS access keys

In order to authenticate to AWS infrastructure from a connection, an identity is required which has the necessary permissions to interact with the target resource. For this tutorial, we will keep it simple and create one IAM user with the policies required to manage both AWS SNS/SQS and DynamoDB. To ensure least privilege access, it is recommended to create a specific identity per connection, however for the sake of brevity, we will create one user which has all three necessary permissions:

• AmazonSNSFullAccess
• AmazonSQSFullAccess
• AmazonDynamoDBFullAccess

To create the IAM User:

1. Navigate to the IAM service in your AWS account
2. Select Users under Access Management > Click Create user
3. Provide the name catalyst-connections
4. Set permissions using the option Attach policies directly
5. Select the three policies listed above > Click Create user
6. Select the newly created user > Select the Security credentials tab
7. Scroll down to the section Access keys > Select Create access key
8. Select Other for use case > Click Next
9. Select Create access key
10. Store the output as environment variables using the below command

export CONNECTION_ACCESS_KEY=<your-access-key>
export CONNECTION_SECRET_KEY=<your-secret-key>

Create connections

Now that we have retrieved the appropriate authentication credentials, we can create the AWS infrastructure connections. There are several ways to create connections as described in Connect to external infrastructure. In this tutorial, we will use the non-prompt based CLI experience.

Create the AWS SNS/SQS Pub/Sub Connection:

 diagrid connection create aws-pubsub --type pubsub.aws.snssqs --metadata accessKey=$CONNECTION_ACCESS_KEY --metadata secretKey=$CONNECTION_SECRET_KEY --metadata region=$AWS_DEFAULT_REGION --scopes inbound,outbound 

Create the AWS DyanmoDB KV Connection:

 diagrid connection create aws-kvstore --type state.aws.dynamodb --metadata accessKey=$CONNECTION_ACCESS_KEY --metadata secretKey=$CONNECTION_SECRET_KEY --metadata region=$AWS_DEFAULT_REGION --metadata table="orders" --scopes inbound 

Recommendation: To ensure the connections were successfully created run diagrid connection list

Create topic subscription

To configure the outbound app to receive messages published by the inbound app using the Pub/Sub API, create a topic subscription targeting the aws-pubsub connection with the following details:

• Subscription name: aws-subscription
• Target Pub/Sub connection: aws-pubsub
• Topic on which to listen for incoming messages: orders
• Default route on the outbound app for message delivery: /pubsub/neworders
• Subscribing App IDs (scopes): outbound

diagrid subscription create aws-subscription --connection aws-pubsub --topic orders --route /pubsub/neworders --scopes outbound

Recommendation: To ensure the subscription was successfully created run diagrid subscription list

Deploy containerized applications to AWS App Runner

To test Catalyst APIs which require inbound calls to Catalyst and outbound calls to an application i.e. Pub/Sub and Request/Reply, you will deploy two sample applications:

• An inbound app for making calls to the Catalyst APIs
• An outbound app for receiving calls from the Catalyst APIs

To make API calls from the application code to Catalyst, the inbound application uses the OSS Dapr SDKs. Environment variables will be configured to enable the Dapr Client to connect to Catalyst instead of a locally-running Dapr sidecar.

Learn more about using the Dapr SDKs to access Catalyst APIs here.

The applications have already been containerized and pushed to a public registry. The sample docker images are available for Python, JavaScript, .NET and Java. You can choose to use any combination of languages for the inbound and outbound apps.

Deploy inbound application

Run the below command to set the coding language you want to deploy for the inbound application:

.NET

export INBOUND_LANGUAGE=csharp

Python

export INBOUND_LANGUAGE=python

JavaScript

export INBOUND_LANGUAGE=javascript

Java

export INBOUND_LANGUAGE=java

Retrieve the HTTP and gRPC endpoints for your Catalyst project. These endpoints will be used by the inbound application to make calls to the Catalyst APIs.

export DAPR_HTTP_ENDPOINT=$(diagrid project get -o json | jq -r '.status.endpoints.http.url')
export DAPR_GRPC_ENDPOINT=$(diagrid project get -o json | jq -r '.status.endpoints.grpc.url')

Recommended: Confirm successful retrieval by running echo $DAPR_HTTP_ENDPOINT && echo $DAPR_GRPC_ENDPOINT

Deploy the inbound application to App Runner. This command will:

• Deploy the sample application in the language of your choice to App Runner
• Configure the environment variables required for the application to invoke Catalyst API calls
• Expose the App Runner instance on a publically-accessible endpoint

aws apprunner create-service --service-name inbound \
--source-configuration '{
    "ImageRepository": {
        "ImageIdentifier": "public.ecr.aws/diagrid/'${INBOUND_LANGUAGE}'-sample:latest",
        "ImageRepositoryType": "ECR_PUBLIC",  
        "ImageConfiguration": { "RuntimeEnvironmentVariables": {
            "DAPR_HTTP_ENDPOINT": "'${DAPR_HTTP_ENDPOINT}'",
            "DAPR_API_TOKEN" : "'${INBOUND_API_TOKEN}'",
            "DAPR_GRPC_ENDPOINT" : "'${DAPR_GRPC_ENDPOINT}'",
            "PUBSUB_NAME": "aws-pubsub", 
            "KVSTORE_NAME": "aws-kvstore", 
            "INVOKE_APPID": "outbound"
        }, "Port": "5001" }},
    "AutoDeploymentsEnabled": false }' \
--instance-configuration '{ "Cpu": "512", "Memory": "1024"}' \
--network-configuration '{"EgressConfiguration": { "EgressType": "DEFAULT" }, "IngressConfiguration": {"IsPubliclyAccessible": true}}'

Important

It may take up to 5 minutes for the App Runner deployment to successfully complete.

Deploy outbound application

Run the below command to set the coding language you want to deploy for the outbound application:

.NET

export OUTBOUND_LANGUAGE=csharp

Python

export OUTBOUND_LANGUAGE=python

JavaScript

export OUTBOUND_LANGUAGE=javascript

Java

export OUTBOUND_LANGUAGE=java

Deploy the outbound application to App Runner. This command will:

• Deploy the sample application in the language of your choice to App Runner
• Expose the App Runner instance on a publically-accessible endpoint

aws apprunner create-service --service-name outbound \
--source-configuration '{
    "ImageRepository": {
        "ImageIdentifier": "public.ecr.aws/diagrid/'${OUTBOUND_LANGUAGE}'-sample:latest",
        "ImageRepositoryType": "ECR_PUBLIC",  
        "ImageConfiguration": { "Port": "5001" }},
    "AutoDeploymentsEnabled": false }' \
--instance-configuration '{ "Cpu": "512", "Memory": "1024"}' \
--network-configuration '{"EgressConfiguration": { "EgressType": "DEFAULT" }, "IngressConfiguration": {"IsPubliclyAccessible": true}}'

Important

It may take up to 5 minutes for the App Runner deployment to successfully complete.

Update the outbound App ID with an app endpoint

Retrieve the FQDN for the outbound application and associate it with the outbound App ID to ensure incoming requests and events from the Catalyst APIs are sent to the running application.

export OUTBOUND_APP_ENDPOINT=$(aws apprunner list-services | jq -r '.ServiceSummaryList[] | select(.ServiceName=="outbound") | .ServiceUrl')

Recommended: Run echo $OUTBOUND_APP_ENDPOINT to ensure the FQDN was successfully retrieved.

To ensure incoming App ID requests and events can be forwarded to the outbound application, update the outbound App ID with the outbound application endpoint and wait for the operation to complete.

diagrid appid update outbound --app-endpoint https://$OUTBOUND_APP_ENDPOINT -w

Interact with the Catalyst APIs

You've now successfully deployed the application code and all of the necessary Catalyst resources to begin invoking the APIs.

Retrieve the App Runner URL or the inbound app which will be used to send requests.

export INBOUND_APP_ENDPOINT=$(aws apprunner list-services | jq -r '.ServiceSummaryList[] | select(.ServiceName=="inbound") | .ServiceUrl')

Recommended: Run echo $INBOUND_APP_ENDPOINT to ensure the FQDN was successfully retrieved.

Use the below commands to kick off API calls from the inbound application to the Catalyst APIs.

Publish/Subscribe

curl -i -X POST https://$INBOUND_APP_ENDPOINT/pubsub/orders  -H "Content-Type: application/json" -d '{"orderId":1}'

Request/Reply

curl -i -X POST https://$INBOUND_APP_ENDPOINT/invoke/orders  -H "Content-Type: application/json" -d '{"orderId":1}'

Key/Value

Save a key/value pair

curl -i -X POST https://$INBOUND_APP_ENDPOINT/kv/orders  -H "Content-Type: application/json" -d '{"orderId":1}'

Update the value of orderId in the data payload with a new key to store another key/value pair.

Retrieve an existing key/value pair

curl -i -X GET https://$INBOUND_APP_ENDPOINT/kv/orders/1

Delete an existing key/value pair

curl -i -X DELETE https://$INBOUND_APP_ENDPOINT/kv/orders/1

Validate APIs via logs

To validate the APIs are woring as expected, use the commands below to inspect your application logs in a secondary terminal as you make requests.

App logs

Use the following commands to identify the log groups for each App Runner application and stream the logs:

# Set the inbound instance ID
export INBOUND_ID=$(aws logs describe-log-groups --log-group-name-prefix '/aws/apprunner' | jq -r '.logGroups[].logGroupName' | grep 'inbound.*application' | cut -d'/' -f 5)

# Tail the logs for inbound service
aws logs tail /aws/apprunner/inbound/$INBOUND_ID/application --follow --format json

# On a separate terminal, set the outbound instance ID
export OUTBOUND_ID=$(aws logs describe-log-groups --log-group-name-prefix '/aws/apprunner' | jq -r '.logGroups[].logGroupName' | grep 'outbound.*application' | cut -d'/' -f 5)

# Tail the logs for outbound service
aws logs tail /aws/apprunner/outbound/$OUTBOUND_ID/application --follow --format json

App ID logs

To view the App ID logs for both the inbound and outbound applications run the following command

diagrid appid logs -all --tail 5 --follow

Summary

In this tutorial, you were able to instrument two applications with the Catalyst APIs to achieve streamlined communication and easy onboarding of supporting infrastructure services, all while simplifying the application code and achieving a greater level of development consistency.

Clean up resources

Once you're done testing the APIs, you can use these commands to clean up the AWS and Catalyst resources you created in this tutorial.

Delete App Runner instances

aws apprunner delete-service --service-arn=`aws apprunner list-services | jq -r '.ServiceSummaryList[] | select(.ServiceName=="inbound") | .ServiceArn'`
aws apprunner delete-service --service-arn=`aws apprunner list-services | jq -r '.ServiceSummaryList[] | select(.ServiceName=="outbound") | .ServiceArn'`

Delete DynamoDB table

Delete the DynamoDB orders table.

aws dynamodb delete-table --table-name=orders

Delete SQS queue, SNS topic, and subscription

export TOPIC_ARN=$(aws sns list-topics --query "Topics[?ends_with(TopicArn, 'orders')].TopicArn | [0]" --output text)
export SUBSCRIPTION_ARN=$(aws sns list-subscriptions-by-topic --topic-arn=$TOPIC_ARN | jq -r '.Subscriptions[0].SubscriptionArn')
export QUEUE_URL=$(aws sqs list-queues --queue-name-prefix=outbound | jq -r '.QueueUrls[0]')

aws sns unsubscribe --subscription-arn=$SUBSCRIPTION_ARN
aws sns delete-topic --topic-arn=$TOPIC_ARN
aws sqs delete-queue --queue-url=$QUEUE_URL

Delete Catalyst resources

diagrid appid delete inbound 
diagrid appid delete outbound
diagrid connection delete aws-pubsub
diagrid connection delete aws-kvstore 

If you want to delete the entire Catalyst project, including the managed infrastructure resources, run the diagrid project delete command.

diagrid project delete catalyst-project