# AWS Deployment

This guide walks through deploying a Catalyst Enterprise Self-Hosted region into a private AWS VPC. It provisions an EKS cluster, RDS PostgreSQL, and a Bastion host, then installs the Catalyst Helm chart against the resulting cluster.

:::warning Demonstration only

This guide is intended as a reference walkthrough for demonstration and proof-of-concept deployments. It is not a production-ready deployment recipe — use the [Production Planning](./production-planning.mdx) guidance and adapt this Terraform to your organization's networking, security, and operational requirements before going to production.

:::

:::note Operating system

The commands in this guide are written for Linux or macOS. Adaptations may be required on Windows.

:::

## Architecture overview

The Terraform in this guide provisions a private VPC and deploys the Catalyst data plane to an EKS cluster inside it. Diagrid Cloud manages the Catalyst control plane over an mTLS connection from inside the VPC.

```mermaid
---
title: AWS reference architecture
---
flowchart LR
  Internet((Public<br/>Internet)):::pub

  subgraph Diagrid["<b style='color:#22613f'>Diagrid Cloud</b>"]
    CP("Catalyst<br/>control plane")
  end

  subgraph AWS["<b style='color:#ed8936'>AWS account</b>"]
    direction TB
    R53("Route 53<br/>+ cert-manager<br/>/ Let's Encrypt"):::customerNode
    subgraph VPC["Private VPC"]
      direction TB
      NLB("Network Load<br/>Balancer"):::customerNode
      subgraph EKS["EKS cluster — Catalyst data plane"]
        direction LR
        DAPR("Dapr<br/>deployment")
        AGENT("Catalyst<br/>agent")
      end
      Bastion("Bastion<br/>host (EC2)"):::customerNode
      RDS[("RDS PostgreSQL<br/>workflow store")]:::customerNode
    end
  end

  Internet--"TLS"-->NLB
  R53-.->NLB
  NLB-->EKS
  AGENT<--"mTLS"-->CP
  DAPR-->RDS
  Bastion-->EKS

  style Diagrid stroke:#22613f,stroke-width:2px
  style AWS stroke:#ed8936,stroke-width:2px
  style EKS stroke-dasharray:4 3,stroke-width:1.5px
  classDef customerNode stroke:#ed8936,stroke-width:2px
  classDef pub stroke:#999,stroke-width:1px
```

## Prerequisites

- [Diagrid CLI](../../../references/catalyst/catalyst-cli.mdx)
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), configured with credentials for your AWS account
- [Terraform](https://developer.hashicorp.com/terraform/install) and `make` (used to provision the AWS infrastructure)
- [Helm](https://helm.sh/)
- [jq](https://stedolan.github.io/jq/download/)
- An AWS account with permissions to create VPC, EKS, EC2, IAM, and related resources
- **Four free Elastic IPs in the AWS region you deploy to.** The stack uses one for each NAT gateway (three, one per availability zone) and one for the Bastion host. The AWS default limit is five. If anything else in the account uses Elastic IPs in that region, check the quota before you apply and ask for an increase:

  ```bash
  aws service-quotas get-service-quota --service-code ec2 --quota-code L-0263D0A3 --region <region>
  ```

  If you run out, the error is easy to miss. Terraform allocates the addresses late in the apply. The `AddressLimitExceeded` errors scroll past, and the error you're left with is an EKS node group that never reaches `ACTIVE`, with `Instances failed to join the kubernetes cluster`. The real cause is that the NAT gateways never got addresses, so the private subnets had no route out. Check the quota instead of debugging the node group.

The Terraform that provisions the AWS infrastructure used below is available in the [`guides/aws`](https://github.com/diagridio/charts/tree/main/guides/aws) directory of the [`diagridio/charts`](https://github.com/diagridio/charts) repository. Clone the repository and run `make` from inside that directory:

```bash
git clone https://github.com/diagridio/charts.git
cd charts/guides/aws
```

## 1. Create a Catalyst region

Use the [Diagrid CLI](../../../references/catalyst/catalyst-cli.mdx) to register a new region. The wildcard ingress domain is configured later, once the Catalyst gateway Load Balancer is available.

```bash
diagrid login

# Create a new region (the ingress domain is updated later)
export JOIN_TOKEN=$(diagrid region create my-aws-region \
  --ingress placeholder.example.com | jq -r .joinToken)

# Create an API key the Diagrid CLI can use later from inside the VPC
# (--duration is in seconds; 604800 = 7 days)
export API_KEY=$(diagrid apikey create \
  --name aws-key \
  --role cra.diagrid:editor \
  --duration 604800 | jq -r .token)
```

:::note

Keep the values of `JOIN_TOKEN` and `API_KEY` at hand — steps 8 and 11 use them **on the Bastion host**, where these environment variables are not set. Print them with `echo $JOIN_TOKEN` / `echo $API_KEY` when you need to copy them over.

If `diagrid apikey create` fails with `Maximum number of apikeys limit reached`, list and remove unused keys with `diagrid apikey list` and `diagrid apikey delete <id>`. Similarly, `diagrid region create` fails if a region named `my-aws-region` already exists.

:::

## 2. Deploy AWS resources

Use the provided Terraform to deploy the required infrastructure to your AWS account. This provisions an EKS cluster in a private VPC alongside a Bastion host that you can use to securely access the cluster for admin operations.

```bash
# Authenticate with AWS and confirm the active profile
aws sts get-caller-identity

# Initialize Terraform
make init

# Show the Terraform plan
make plan

# Apply the Terraform plan
make apply
```

:::tip

If `make plan` fails with `Your query returned no results` for the bastion AMI, the pinned Amazon Linux 2023 AMI has been deregistered by AWS. Override it with a currently available AMI without editing the Terraform:

```bash
TF_VAR_bastion_specific_ami_name=$(aws ec2 describe-images --owners amazon \
  --filters "Name=name,Values=al2023-ami-2023.*-kernel-6.*-x86_64" \
  --query 'reverse(sort_by(Images,&CreationDate))[0].Name' --output text) make plan
```

:::

## 3. Connect to the Bastion host

Connect to the Bastion host via [EC2 Instance Connect](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Connect-using-EC2-Instance-Connect.html):

```bash
EC2_CONNECT_CMD=$(make output bastion_ec2_instance_connect_command)

# Execute the EC2 connect command
eval $EC2_CONNECT_CMD
```

The Bastion host resides within the private VPC and can communicate with the EKS cluster. It comes with `kubectl`, Helm, the AWS CLI, and the Diagrid CLI pre-installed — the later steps rely on these.

:::note

By default the bastion accepts SSH from anywhere (`0.0.0.0/0`). Restrict it to your network with `make apply BASTION_ALLOWED_CIDR=<your-cidr>`.

:::

## 4. Configure Kubernetes access

Configure the pre-installed `kubectl` on the Bastion host to talk to your EKS cluster.

On your local machine, read the EKS cluster name and region from the Terraform outputs:

```bash
make output eks_cluster_name
make output eks_cluster_region
```

On the Bastion host SSH session, configure `kubectl`:

```bash
export EKS_CLUSTER_NAME="<value-from-eks_cluster_name-output>"
export EKS_CLUSTER_REGION="<value-from-eks_cluster_region-output>"

aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $EKS_CLUSTER_REGION

kubectl config current-context
```

### Set the default storage class

EKS creates a `gp2` storage class and nothing else. Do not make it the cluster default. `gp2` binds a volume immediately, before the pod that will use it is scheduled, so a volume can be placed in an availability zone that holds no worker node — and an EBS volume is only mountable from its own zone, so that volume can never be mounted again. Your cluster spans three zones and starts with two worker nodes, which is exactly the shape in which this happens. `gp2` also scales IOPS with volume size, at 3 per GiB, so a small volume is starved: 10 GiB buys 100 IOPS.

Create a `gp3` class instead and make it the default. `gp3` binds on first consumer, so a volume follows its pod into a zone the pod can run in, and it has a 3000 IOPS floor at any size. The EBS CSI driver that provisions it is already installed, as an EKS add-on, by the Terraform you applied in step 2.

Apply the new class first, so the cluster is never left without a default:

```bash
kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
  iops: "3000"
  throughput: "125"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
EOF
```

Then remove the default marker from `gp2`:

```bash
kubectl patch storageclass gp2 \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
```

Between the two commands the cluster briefly has two default classes. Kubernetes warns and takes the newest, which is `gp3`. Running the commands the other way round leaves a window with no default at all, and a volume claim created in that window stays unbound permanently.

Confirm that `gp3` is now the only default:

```bash
kubectl get storageclass
```

`gp3` carries `(default)` in the output and `gp2` does not.

### Optional: Install the krew kubectl plugin manager

Install [krew](https://krew.sigs.k8s.io/docs/user-guide/setup/install/) on the Bastion host for quality-of-life utilities like `kubectx` and `kubens`:

```bash
sudo yum install git --assumeyes
(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

cat >> ~/.bashrc << EOF
export PATH="\${KREW_ROOT:-\$HOME/.krew}/bin:\$PATH"
EOF
```

Restart your shell so the `PATH` change takes effect, then install `kubectx`, `kubens`, and a small alias file:

```bash
kubectl krew install ctx
kubectl krew install ns

cat >> ~/.bashrc << EOF
source ~/.alias
EOF

cat > ~/.alias << EOF
alias k='kubectl'
alias kp='kubectl get pods'
alias kpa='kubectl get pods --all-namespaces'
alias kf='kubectl logs -f'
alias kx='k ctx'
alias kns='k ns'
EOF
```

## 5. Install the AWS Load Balancer controller

On your local machine, read the cluster name and the controller IAM role ARN from the Terraform outputs:

```bash
make output eks_cluster_name
make output aws_load_balancer_controller_role_arn
```

On the Bastion host SSH session, install the controller via Helm:

```bash
helm repo add eks https://aws.github.io/eks-charts
helm repo update eks

helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
    --namespace kube-system \
    --set clusterName=<value-from-eks_cluster_name> \
    --set serviceAccount.create=true \
    --set serviceAccount.name=aws-load-balancer-controller \
    --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=<value-from-aws_load_balancer_controller_role_arn-output>
```

## 6. Install cert-manager

On your local machine, read the cert-manager IAM role ARN from the Terraform outputs:

```bash
make output cert_manager_role_arn
```

On the Bastion host SSH session, install cert-manager via Helm:

```bash
helm repo add jetstack https://charts.jetstack.io --force-update

helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version 1.21.0 \
  --set crds.enabled=true \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=<value-from-cert_manager_role_arn-output>
```

### Install a Let's Encrypt ClusterIssuer

Create a `ClusterIssuer` that uses the [DNS-01 challenge](https://cert-manager.io/docs/configuration/acme/dns01/) via Route53. Pass the cert-manager IAM role ARN read above, use your own email address (Let's Encrypt sends expiry notices there), and make sure `region` matches the AWS region you deployed to (`make output eks_cluster_region`):

```bash
cat > ~/lets-encrypt-cluster-issuer.yaml << EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: <your-email-address>
    privateKeySecretRef:
      name: letsencrypt
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        route53:
          region: us-west-2
          role: <value-from-cert_manager_role_arn-output>
          auth:
            kubernetes:
              serviceAccountRef:
                name: cert-manager
EOF

kubectl apply -f ~/lets-encrypt-cluster-issuer.yaml
```

With the `serviceAccountRef` above, cert-manager creates a token for its own service account and exchanges it with AWS STS. To do that, it needs permission to create `serviceaccounts/token`. The cert-manager chart doesn't grant this, so add it yourself:

```bash
cat > ~/cert-manager-token-creator.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cert-manager-token-creator
  namespace: cert-manager
rules:
- apiGroups: [""]
  resources: ["serviceaccounts/token"]
  verbs: ["create"]
  resourceNames: ["cert-manager"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cert-manager-token-creator
  namespace: cert-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cert-manager-token-creator
subjects:
- kind: ServiceAccount
  name: cert-manager
  namespace: cert-manager
EOF

kubectl apply -f ~/cert-manager-token-creator.yaml
```

Without this permission, the DNS-01 challenge stays `pending` forever and the certificate is never issued. You only see the error in the Challenge's events:

```
error getting service account token: failed to request token for cert-manager/cert-manager:
serviceaccounts "cert-manager" is forbidden: User "system:serviceaccount:cert-manager:cert-manager"
cannot create resource "serviceaccounts/token" in API group "" in the namespace "cert-manager"
```

Without the wildcard certificate, the `catalyst-gateway` pod never leaves `Init`, so nothing from step 9 on works. If you add the Role after cert-manager is already running, restart cert-manager so it tries again: `kubectl -n cert-manager rollout restart deployment cert-manager`.

### Optional: provision an ECDSA CA for Dapr Sentry

Dapr Sentry generates its own **Ed25519** root CA by default, and AWS IAM Roles Anywhere only accepts RSA or ECDSA trust anchors. If you plan to [authenticate AWS components with IAM Roles Anywhere](#authenticate-aws-components-with-iam-roles-anywhere), give Sentry an ECDSA CA instead — cert-manager is already installed, so this is a good place to do it.

:::caution Do this before the region joins

The agent submits the Dapr CA to Diagrid Cloud when the region first joins, in step 8. Changing the CA afterwards makes the agent's next join fail with `join token ... was already consumed ... with a different CA trust anchor`, and refreshing the published trust anchors requires a region rejoin from [Diagrid support](mailto:support@diagrid.io).

:::

Install trust-manager alongside cert-manager on the Bastion host SSH session:

```bash
helm upgrade trust-manager oci://quay.io/jetstack/charts/trust-manager \
  --install \
  --namespace cert-manager
```

Create a self-signed ECDSA root, an issuing CA for Sentry, and a `Bundle` that publishes the root:

```bash
cat > ~/dapr-pki.yaml << EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: internal-ca
  secretName: internal-ca
  privateKey:
    algorithm: ECDSA
  # cert-manager would otherwise default this root to 90 days. This is the
  # certificate you register as the Roles Anywhere trust anchor, and AWS stores
  # a copy of it, so give it a long lifetime. The Sentry intermediate below
  # rotates yearly underneath it without invalidating the anchor.
  duration: 87600h  # 10 years
  renewBefore: 720h # renew 30 days before
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: internal-ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: internal-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dapr-trust-bundle
  namespace: cert-manager
spec:
  # The issuer's subject must be non-empty. Workload leaf certificates take
  # their Issuer DN from this certificate's subject, and IAM Roles Anywhere
  # identifies a certificate by issuer + serial — it rejects a leaf whose
  # issuer DN is empty with "Unable to form certificate ID".
  commonName: dapr-sentry
  uris:
    - "spiffe://prj-root.trust.diagrid.io/ns/root-dapr-system/dapr-sentry"
  isCA: true
  usages:
    - digital signature
    - key encipherment
    - cert sign
    - crl sign
  privateKey:
    algorithm: ECDSA
    size: 256
  duration: 8760h   # 1 year
  renewBefore: 720h # renew 30 days before
  secretName: dapr-trust-bundle
  issuerRef:
    name: internal-ca-issuer
    kind: Issuer
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: dapr-trust-bundle
spec:
  sources:
  - secret:
      name: dapr-trust-bundle
      key: ca.crt
  target:
    configMap:
      key: ca.crt
    namespaceSelector:
      matchLabels:
        name: cert-manager
EOF

kubectl apply -f ~/dapr-pki.yaml
```

The bundle you publish must contain only self-signed root certificates — Diagrid Cloud rejects a trust anchor that contains an intermediate. Here `ca.crt` is the self-signed `internal-ca` root, and Sentry's own `dapr-trust-bundle` certificate is the intermediate that signs workload identities, so the chain validates against the root you register with AWS.

You wire these secrets into Catalyst in step 8. See the [Dapr PKI guide](https://github.com/diagridio/charts/tree/main/guides/dapr-pki) and the [Helm chart reference](./reference.md#dapr-pki) for the full set of options.

## 7. Install monitoring tools

These are recommended utilities for observing the cluster and Catalyst components. They are not strictly required for Catalyst to function.

### Metrics server

```bash
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/

helm upgrade --install metrics-server metrics-server/metrics-server
```

### Prometheus node exporter

```bash
cat > ~/prometheus-node-exporter-config.yaml << EOF
podAnnotations:
  prometheus.io/path: /metrics
  prometheus.io/port: "9100"
  prometheus.io/scrape: "true"
rbac:
  pspEnabled: false
resources:
  limits:
    memory: 50Mi
  requests:
    cpu: 10m
    memory: 25Mi
EOF

helm upgrade --install prometheus-node-exporter \
  oci://ghcr.io/prometheus-community/charts/prometheus-node-exporter \
  --values prometheus-node-exporter-config.yaml
```

### kube-state-metrics

```bash
cat > ~/kube-state-metrics-config.yaml << EOF
podAnnotations:
  prometheus.io/port: "8080"
  prometheus.io/scrape: "true"
EOF

helm upgrade --install kube-state-metrics \
  oci://ghcr.io/prometheus-community/charts/kube-state-metrics \
  --values kube-state-metrics-config.yaml
```

## 8. Configure and install Catalyst

:::info Secrets backend

This example uses the default Kubernetes Secrets backend. Catalyst can also be configured to use AWS Secrets Manager — see the [Helm Chart Reference](./reference.md) for the full set of options.

:::

Read the RDS PostgreSQL connection details from the Terraform outputs:

```bash
make output postgresql_endpoint
make output postgresql_master_user_secret_arn
make output eks_cluster_region

# Fetch the master user password from AWS Secrets Manager
aws --region $(make output eks_cluster_region) secretsmanager get-secret-value \
  --secret-id $(make output postgresql_master_user_secret_arn) \
  --query SecretString --output text | jq -r '.password'

# Print the join token from step 1 so you can copy it to the Bastion host
echo $JOIN_TOKEN
```

On the Bastion host SSH session, create a Helm values file that wires Catalyst up to RDS and exposes the gateway via an AWS Network Load Balancer (NLB). The `JOIN_TOKEN` from step 1 is not set in the Bastion shell — export it with the value printed above:

```bash
export RDS_POSTGRESQL_ENDPOINT="<value-from-postgresql_endpoint-output>"
export RDS_POSTGRESQL_PASSWORD="<value-from-aws-cli>"
export JOIN_TOKEN="<value-from-step-1>"

cat > catalyst-values.yaml << EOF
agent:
  config:
    project:
      default_managed_state_store_type: postgresql-shared-external
      external_postgresql:
        enabled: true
        auth_type: connectionString
        namespace: postgresql
        connection_string_host: $RDS_POSTGRESQL_ENDPOINT
        connection_string_port: 5432
        connection_string_username: postgres
        connection_string_password: "$RDS_POSTGRESQL_PASSWORD"
gateway:
  tls:
    enabled: true
    secretName: "cert-wildcard"
  envoy:
    service:
      type: LoadBalancer
      httpsPort: 443
      httpsTargetPort: 8443
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "tcp"
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
EOF

helm install catalyst oci://public.ecr.aws/diagrid/catalyst \
     -n cra-agent \
     --create-namespace \
     -f catalyst-values.yaml \
     --set join_token="${JOIN_TOKEN}"
```

:::note Using your own Dapr CA

If you provisioned an ECDSA CA in [step 6](#optional-provision-an-ecdsa-ca-for-dapr-sentry), point Sentry at it by adding this stanza under `agent.config` in `catalyst-values.yaml` before installing — the CA is submitted to Diagrid Cloud during this install and cannot be swapped afterwards without a region rejoin:

```yaml
agent:
  config:
    internal_dapr:
      pki:
        issuer:
          secret:
            name: dapr-trust-bundle
            namespace: cert-manager
            cert: tls.crt
            key: tls.key
        trust:
          config_map:
            name: dapr-trust-bundle
            namespace: cert-manager
            chain: ca.crt
```

:::

## 9. Configure ingress and a wildcard TLS certificate

Create a Route53 hosted zone and wildcard TLS certificate for a domain you own. This allows applications to reach the Catalyst gateway over a friendly DNS name and TLS.

:::note

This step is required for the rest of the guide: the Catalyst gateway pod stays unready until the certificate below is issued, and the Diagrid CLI commands in step 11 (`diagrid listen`, `diagrid call`) connect through the region's ingress domain, so they only work once DNS delegation and the certificate are in place.

:::

```bash
REGION_INGRESS_ENDPOINT="<somename>.<domain that you own>" make apply

make output region_ingress_endpoint

# Update the region with the wildcard ingress domain
diagrid region update my-aws-region \
  --ingress "<value-from-region_ingress_endpoint-output>"
```

Delegate the subdomain to Route53 by adding NS records in your domain registrar's DNS settings that point to the Route53 hosted zone's nameservers. Once delegation propagates, `dig` should resolve the NLB:

```bash
dig whatever.<somename>.<domain that you own>
```

### Issue a Let's Encrypt wildcard certificate

On your local machine:

```bash
make output region_wildcard_domain
```

On the Bastion host SSH session:

```bash
cat > ~/wildcard-certificate.yaml << EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cert-wildcard
  namespace: cra-agent
spec:
  dnsNames:
  - "<value-from-region_wildcard_domain-output>"
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt
  secretName: cert-wildcard
  usages:
  - digital signature
  - key encipherment
EOF

kubectl apply -f ~/wildcard-certificate.yaml
```

## 10. Verify the installation

Wait for all Catalyst pods to become ready. This can take several minutes.

```bash
kubectl -n cra-agent wait --for=condition=ready pod --all --timeout=5m
```

:::note

Until the wildcard certificate from step 9 is issued, the `catalyst-gateway` pod stays in `Init` with `FailedMount` warnings for the `cert-wildcard` secret — this is expected and resolves on its own once the certificate is ready. Also note that `kubectl wait` reports pods sequentially: a single unready pod makes it print `timed out` for every pod listed after it, even ones that are already ready. Use `kubectl -n cra-agent get pods` to see the actual per-pod state.

:::

## 11. Create a project and deploy apps

On your local machine, create a project in the new region and a couple of apps to test connectivity:

```bash
diagrid project create aws-project --region my-aws-region --use

diagrid app create app1
diagrid app create app2

# Wait until the apps are ready
diagrid app list
```

Because the Catalyst gateway is internal to the VPC, test invocation from within the VPC. Open a **new terminal window** on your local machine and connect to the Bastion host again to start a listener:

```bash
# Print the API key from step 1 so you can copy it to the Bastion host
echo $API_KEY

EC2_CONNECT_CMD=$(make output bastion_ec2_instance_connect_command)
eval $EC2_CONNECT_CMD

# On the new Bastion host SSH session, log in with the API key from step 1
# (the API_KEY environment variable is not set in the Bastion shell)
diagrid login --api-key="<value-from-step-1>"
diagrid project use aws-project

# The Diagrid CLI can create a listener for an app and print any requests
# sent to it. Wait until you see:
# ✅ Connected App ID "app1" to http://localhost:<port> ⚡️
diagrid listen -a app1
```

From the original Bastion session, send a service-invocation request between apps:

```bash
# Call app1 from app2 via the internal gateway using the wildcard domain
diagrid call invoke get app1.hello -a app2
```

You should see the request received (`'method': 'GET', 'url': '/hello'`) on the listener. This proves that you can use [Dapr's service invocation API](https://docs.dapr.io/developing-applications/building-blocks/service-invocation/service-invocation-overview/) by calling an app via the internal NLB using the Catalyst region's wildcard domain.

To open the Catalyst web console for the project:

```bash
diagrid web
```

## 12. Write your applications

Now that the Catalyst region is up and a project with two apps is in place, head over to the [Connect to Catalyst guide](https://docs.diagrid.io/develop/connect) to start building applications that use these apps. Applications running inside the VPC can reach the internal gateway directly at `http://<GATEWAY_HOSTNAME>:8080`.

For applications running outside the VPC, you must own a domain that can route traffic to your Catalyst gateway, configure TLS on the gateway (already done in step 8: `gateway.tls.enabled: true` with the `cert-wildcard` secret issued in step 9), and ensure your applications trust the root CA for the certificate you use. Then point your applications at the Catalyst region with the standard Dapr environment variables:

- `DAPR_GRPC_ENDPOINT=<your-project-grpc-url>`
- `DAPR_HTTP_ENDPOINT=<your-project-http-url>`
- `DAPR_API_TOKEN=<your-app-api-token>`

## Tear the region down

Running `make destroy` on its own doesn't work. It fails partway through and leaves resources running, and you keep paying for them.

Terraform doesn't create the gateway's Network Load Balancer. The Catalyst chart asks for a `LoadBalancer` Service, and the AWS Load Balancer Controller, which runs inside the cluster, creates the load balancer and its security groups. Terraform doesn't know about any of them. So `terraform destroy` deletes the EKS cluster, and the controller that would have cleaned up goes with it. The load balancer stays behind, holding network interfaces in the subnets Terraform is trying to delete:

```
Error: deleting EC2 Subnet (subnet-...): DependencyViolation:
       The subnet '...' has dependencies and cannot be deleted.
Error: deleting EC2 Internet Gateway (igw-...): detaching ...:
       DependencyViolation: Network vpc-... has some mapped public address(es).
```

Uninstall the chart first. This deletes the Service, and the controller then deletes the load balancer and its security groups while it's still running:

```bash
# 1. Remove the Kubernetes objects that own AWS resources Terraform does not manage.
helm uninstall catalyst -n cra-agent
kubectl -n cra-agent wait --for=delete svc/gateway-envoy --timeout=5m

# 2. Confirm the load balancer is gone before continuing.
make output vpc_id
aws elbv2 describe-load-balancers --region <region> \
  --query "LoadBalancers[?VpcId=='<value-from-vpc_id-output>'].LoadBalancerName"

# 3. Only now destroy the infrastructure.
make destroy REGION=<region> REGION_INGRESS_ENDPOINT=<domain>
```

Check two RDS settings before you destroy. If either is wrong, the destroy fails near the end:

- **Deletion protection.** `postgresql_deletion_protection` is `true` by default, and RDS won't delete a protected instance. Set it to `false` and apply that change on its own before you destroy.
- **Final snapshots.** Unless `postgresql_skip_final_snapshot` is `true`, each instance takes a final snapshot before it's deleted. Snapshot names must be unique in each account and region. If a snapshot with that name already exists, the destroy stops with `DBSnapshotAlreadyExists`, and the database, VPC, and subnets keep running. Delete the old snapshot with `aws rds delete-db-snapshot --region <region> --db-snapshot-identifier <name>`, or set `postgresql_final_snapshot_identifier` to an unused name and apply that first.

:::warning A partial destroy fails quietly and keeps costing money

Nothing warns you that a load balancer, a VPC, a Bastion host, and three unattached Elastic IPs are still there. AWS charges for an unattached Elastic IP by the hour, and it counts against the quota in the prerequisites. So a region you didn't fully tear down can block your next deployment to that AWS region months later. After the destroy, check the region for leftover EKS clusters, RDS instances, EC2 instances, NAT gateways, load balancers, and Elastic IPs.

:::

## Authenticate AWS components with IAM Roles Anywhere

Catalyst apps can authenticate to AWS services — such as DynamoDB, S3, Secrets Manager, SNS/SQS, or RDS — using [AWS IAM Roles Anywhere](https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html) instead of storing long-lived AWS access keys in component metadata. Each app already has a SPIFFE X.509 identity issued by the region's Dapr Sentry; IAM Roles Anywhere lets that certificate be exchanged for short-lived AWS credentials at runtime.

:::note Prerequisites

You need the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured with permission to manage IAM and Roles Anywhere resources, the Diagrid CLI, and an app in a Catalyst project. The examples also use [`jq`](https://github.com/jqlang/jq).

Your region must also be running an **RSA or ECDSA** Dapr CA — see [step 6](#optional-provision-an-ecdsa-ca-for-dapr-sentry). AWS rejects the Ed25519 CA that Dapr Sentry generates by default, so on a region left on that default `create-trust-anchor` fails with `Unsupported key type` and the CA cannot be changed without a region rejoin.

:::

### How it works

- Each app is issued a SPIFFE X.509 SVID by the region's Dapr Sentry, of the form `spiffe://<trust-domain>/ns/prj-<project-uid>/<app-name>`.
- You register the region's certificate authority (CA) as a Roles Anywhere **trust anchor** in your AWS account, so AWS trusts certificates issued by Sentry.
- A Roles Anywhere **profile** links the trust anchor to an IAM **role**.
- The role's trust policy allows the `rolesanywhere.amazonaws.com` service principal to assume it **only** for that app's SPIFFE ID, matched on the `aws:PrincipalTag/x509SAN/URI` condition. This is what binds a specific app to a specific AWS role.
- At runtime the app's sidecar presents its SVID to IAM Roles Anywhere (`CreateSession`) and exchanges it for temporary STS credentials. No static keys are stored anywhere.

The `setup-catalyst-iamra.sh` helper script in [`guides/aws`](https://github.com/diagridio/charts/tree/main/guides/aws) automates the trust anchor, role, and profile for a single app. Run it with `--project` and `--app`, or follow the manual steps below.

```bash
./setup-catalyst-iamra.sh --project <your-project> --app <your-app>
```

It resolves the app's SPIFFE ID, registers the region's CA as a trust anchor, creates the IAM role with the correct trust policy, and creates the profile — then prints the three ARNs you wire into the component.

### Set up IAM Roles Anywhere manually

The following steps reproduce the helper script for one app. The trust anchor (step 3) is created once per region; repeat steps 4–5 for each app that needs its own role.

#### 1. Set environment variables

Use the same AWS region you deployed the infrastructure to (`make output eks_cluster_region`), so the Roles Anywhere resources live alongside the rest of the deployment:

```bash
export CATALYST_PROJECT="aws-project"
export CATALYST_APP="app1"
export AWS_REGION="us-west-2"
```

#### 2. Resolve the app's SPIFFE ID

The role's trust policy is scoped to this exact value:

```bash
SPIFFE_ID=$(diagrid app get "$CATALYST_APP" --project "$CATALYST_PROJECT" -o json | jq -r '.status.appIds[0].status.spiffeId')
echo "$SPIFFE_ID"
# spiffe://<trust-domain>/ns/prj-<project-uid>/app1
```

#### 3. Register the trust anchor

The region object returned by the Diagrid CLI includes the region's CA bundle. Read it and register it as a Roles Anywhere trust anchor:

```bash
# Resolve the region for your project
CATALYST_REGION=$(diagrid project get "$CATALYST_PROJECT" -o json | jq -r '.spec.region')

# Read the region's CA bundle and register it as the trust anchor
diagrid region get "$CATALYST_REGION" -o json | jq -r '.status.trustAnchors' > diagrid-ca.pem

jq -n --arg name "catalyst-${CATALYST_REGION}" --rawfile pem diagrid-ca.pem \
  '{name:$name, source:{sourceType:"CERTIFICATE_BUNDLE", sourceData:{x509CertificateData:$pem}}, enabled:true}' \
  > trust-anchor.json

TRUST_ANCHOR_ARN=$(aws rolesanywhere create-trust-anchor --region "$AWS_REGION" \
  --cli-input-json file://trust-anchor.json \
  --query 'trustAnchor.trustAnchorArn' --output text)
```

The trust anchor holds a copy of the CA certificate, not a reference to it. If the Dapr CA is renewed or replaced, refresh the anchor with the new bundle — otherwise `CreateSession` starts failing once the registered certificate expires:

```bash
diagrid region get "$CATALYST_REGION" -o json | jq -r '.status.trustAnchors' > diagrid-ca.pem

aws rolesanywhere update-trust-anchor --region "$AWS_REGION" \
  --trust-anchor-id "$(aws rolesanywhere list-trust-anchors --region "$AWS_REGION" \
    --query "trustAnchors[?name=='catalyst-${CATALYST_REGION}'].trustAnchorId | [0]" --output text)" \
  --source "$(jq -n --rawfile pem diagrid-ca.pem \
    '{sourceType:"CERTIFICATE_BUNDLE", sourceData:{x509CertificateData:$pem}}')"
```

:::tip

The same CA bundle is also served unauthenticated over HTTPS from the region's trust endpoint, exposed on the region object at `.status.endpoints.trustAnchors`. Resolve the URL once, then any automation can fetch the bundle without the Diagrid CLI. Note that the endpoint is served by the region's own ingress, so whatever fetches it needs network connectivity to the private region, and it only points at the region once the wildcard ingress domain from step 9 is set — before that it falls back to Diagrid's control plane endpoint, which serves a different CA:

```bash
TRUST_ENDPOINT=$(diagrid region get "$CATALYST_REGION" -o json | jq -r '.status.endpoints.trustAnchors')
curl -fsS "$TRUST_ENDPOINT" -o diagrid-ca.pem
```

:::

#### 4. Create the IAM role and trust policy

The trust policy lets Roles Anywhere assume the role only for this app's SPIFFE ID:

```bash
cat > role-trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "rolesanywhere.amazonaws.com" },
    "Action": ["sts:AssumeRole", "sts:TagSession", "sts:SetSourceIdentity"],
    "Condition": {
      "ArnEquals": { "aws:SourceArn": "${TRUST_ANCHOR_ARN}" },
      "StringEquals": { "aws:PrincipalTag/x509SAN/URI": "${SPIFFE_ID}" }
    }
  }]
}
EOF

ROLE_ARN=$(aws iam create-role \
  --role-name "catalyst-${CATALYST_APP}" \
  --assume-role-policy-document file://role-trust.json \
  --query 'Role.Arn' --output text)
```

#### 5. Create the Roles Anywhere profile

```bash
PROFILE_ARN=$(aws rolesanywhere create-profile --region "$AWS_REGION" \
  --name "catalyst-${CATALYST_APP}" \
  --role-arns "$ROLE_ARN" \
  --enabled \
  --query 'profile.profileArn' --output text)
```

You now have the three ARNs every IAM Roles Anywhere component needs: `TRUST_ANCHOR_ARN`, `PROFILE_ARN`, and `ROLE_ARN`. The role grants no access yet — the example below attaches a permissions policy.

### Example: AWS DynamoDB state store

This wires the app up to a DynamoDB table as a Dapr state store.

#### 1. Create the DynamoDB table

The Dapr DynamoDB state store requires a partition key named `key`:

```bash
aws dynamodb create-table --region "$AWS_REGION" \
  --table-name Contracts \
  --attribute-definitions AttributeName=key,AttributeType=S \
  --key-schema AttributeName=key,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST
```

:::note Partition key

The partition key **must** be named `key` (type `S`), with no sort key — this is what the Dapr DynamoDB state store expects. To use a different name, set the component's `partitionKey` metadata field.

:::

#### 2. Grant the role access to the table

Attach a read/write permissions policy to the role created above:

```bash
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

cat > perms.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "dynamodb:GetItem", "dynamodb:BatchGetItem", "dynamodb:Query", "dynamodb:Scan",
      "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:DeleteItem", "dynamodb:BatchWriteItem",
      "dynamodb:TransactGetItems", "dynamodb:TransactWriteItems"
    ],
    "Resource": [
      "arn:aws:dynamodb:${AWS_REGION}:${ACCOUNT_ID}:table/Contracts",
      "arn:aws:dynamodb:${AWS_REGION}:${ACCOUNT_ID}:table/Contracts/index/*"
    ]
  }]
}
EOF

aws iam put-role-policy --role-name "catalyst-${CATALYST_APP}" \
  --policy-name catalyst-access --policy-document file://perms.json
```

#### 3. Create the Catalyst component

Create the component with the `AWS: IAM Roles Anywhere` authentication profile — the three ARNs from the setup steps — and scope it to the app:

```bash
diagrid component create dynamodb \
  --type state.aws.dynamodb \
  --project "$CATALYST_PROJECT" \
  --metadata region="$AWS_REGION" \
  --metadata table=Contracts \
  --metadata assumeRoleArn="$ROLE_ARN" \
  --metadata trustAnchorArn="$TRUST_ANCHOR_ARN" \
  --metadata trustProfileArn="$PROFILE_ARN" \
  --scopes "$CATALYST_APP" \
  --wait
```

Once the component is ready, the app can read and write state through the Dapr state API, and Catalyst authenticates every DynamoDB call with short-lived credentials minted from the app's identity. The same `assumeRoleArn` / `trustAnchorArn` / `trustProfileArn` profile is available on the other AWS components (Secrets Manager, SNS/SQS, S3, RDS/PostgreSQL, and more) — swap the `--type` and the IAM permissions policy for the target service.