# Protect the regional PKI keys When you [bring your own PKI](./system-architecture.md#option-2-bring-your-own-pki), two private keys decide who can mint a workload identity in your region: - **The root CA key.** It signs the issuer certificate. Its certificate is the trust anchor the region registers with the Catalyst control plane when it first joins, so it's long-lived, and changing it later needs a rejoin through [Diagrid support](mailto:support@diagrid.io). - **The issuer key.** Sentry signs every workload certificate with it, so it has to be in the cluster. Its certificate is an intermediate under the root, and you can renew it as often as you like. This page shows how to keep the root key out of the cluster entirely, and how to limit who can read the issuer key, encrypt it and rotate it, with Kubernetes and cloud integrations. ## Where the issuer key is stored The Catalyst agent reads the issuer certificate and key from the Secret you name in `agent.config.internal_dapr.pki.issuer.secret`, and installs the region's Dapr control plane with them. So the key exists in more than one Secret: | Namespace | Secret | Holds | |-----------|--------|-------| | The one you choose, such as `cert-manager` | The one you name, such as `dapr-trust-bundle` | The issuer certificate and key you provide. | | `root-dapr-system` | `sh.helm.release.v1.*` | The Helm release of the Dapr control plane, including the issuer key, one Secret per kept revision. | | `root-dapr-system` | `dapr-trust-bundle` | The issuer certificate and key that Sentry loads, as `issuer.crt` and `issuer.key`. | Everything below applies to all three. Two consequences follow: - **Don't put the key in Helm values.** The inline form, `pki.issuer.cert_pem` and `pki.issuer.key_pem`, adds the key to your values file, your CI system and the Catalyst Helm release. Use a Secret reference. - **Moving the Secret to a locked-down namespace doesn't hide it from the agent.** The agent has cluster-wide read access to Secrets, as the [Helm reference](./reference.md#permissions-rbac) explains. That namespace still keeps people and other workloads away from it. ## Keep the root key out of the cluster The [Dapr PKI guide](https://github.com/diagridio/charts/tree/main/guides/dapr-pki) and the [AWS deployment](./aws-installation-guide.mdx) create the root with cert-manager's self-signed issuer, so the root key sits in a Secret next to the issuer key. Anyone who can read that Secret can mint a new issuer the region trusts, for as long as the root lives. Instead, keep the root in a managed certificate authority and let a cert-manager issuer for it sign the intermediate. The root key then never leaves the managed service, and the cluster holds only the issuer key. ### AWS Private CA Use [AWS Private CA](https://docs.aws.amazon.com/privateca/latest/userguide/PcaWelcome.html) with the [`aws-privateca-issuer`](https://github.com/cert-manager/aws-privateca-issuer) plugin for cert-manager. Diagrid issues the intermediates for its own control plane the same way. 1. Create a root CA in AWS Private CA, or a subordinate CA under it that is allowed to issue CA certificates. The CA's keys are generated and kept in FIPS-validated hardware security modules, and you can't export them. 2. Install `aws-privateca-issuer` in the cluster. Give its service account an IAM role, with IAM Roles for Service Accounts or EKS Pod Identity, that allows `acm-pca:DescribeCertificateAuthority`, `acm-pca:GetCertificate` and `acm-pca:IssueCertificate` on that CA. 3. Create the issuer and the Sentry certificate: ```yaml apiVersion: awspca.cert-manager.io/v1beta1 kind: AWSPCAIssuer metadata: name: dapr-pca namespace: cert-manager spec: arn: arn:aws:acm-pca:::certificate-authority/ region: --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: dapr-trust-bundle namespace: cert-manager spec: commonName: dapr-sentry uris: - "spiffe://prj-root.trust.diagrid.io/ns/root-dapr-system/dapr-sentry" isCA: true usages: - digital signature - key encipherment - cert sign - crl sign privateKey: algorithm: ECDSA size: 256 rotationPolicy: Always duration: 2160h # 90 days renewBefore: 720h # 30 days secretName: dapr-trust-bundle issuerRef: group: awspca.cert-manager.io kind: AWSPCAIssuer name: dapr-pca ``` `rotationPolicy: Always` gives each renewal a new key, so a leaked key stops working when its certificate expires. A shorter `duration` than the one-year example in the Dapr PKI guide shortens that window. 4. Publish the root certificate as the trust anchor. The control plane accepts only self-signed root certificates, so build the ConfigMap from the root itself, not from the `ca.crt` in the issued Secret, which can hold a subordinate CA: ```bash aws acm-pca get-certificate-authority-certificate \ --certificate-authority-arn \ --query Certificate --output text > root-ca.pem kubectl -n cert-manager create configmap dapr-trust-anchors --from-file=ca.crt=root-ca.pem ``` 5. Point the agent at both, before the region first joins: ```yaml agent: config: internal_dapr: pki: issuer: secret: name: dapr-trust-bundle namespace: cert-manager trust: config_map: name: dapr-trust-anchors namespace: cert-manager ``` ### Other certificate authorities The same pattern works with any cert-manager issuer that can sign a CA certificate, such as the [Vault issuer](https://cert-manager.io/docs/configuration/vault/) or the [Google CAS issuer](https://github.com/jetstack/google-cas-issuer): change the `issuerRef`, and publish the root you control as the trust anchor. Azure Key Vault has no cert-manager issuer that signs CA certificates. On Azure, use one of the issuers above, or sign the issuer certificate outside the cluster with a root you keep offline, store it in Key Vault, and sync it into the Secret with [External Secrets Operator](https://external-secrets.io/). The Secrets Store CSI driver on its own doesn't work here: the agent reads a Kubernetes Secret through the API, not a file mounted into a pod. ## Limit who can read the issuer key - **Keep the Secret in a namespace that only cert-manager and your platform team use.** Don't grant `get`, `list` or `watch` on Secrets in that namespace or in `root-dapr-system` to application teams, and avoid ClusterRoles that grant them on all Secrets. - **Check who has access.** For example, to check a group: ```bash kubectl auth can-i get secrets -n cert-manager --as=dev-user --as-group=developers kubectl auth can-i get secrets -n root-dapr-system --as=dev-user --as-group=developers ``` - **Audit reads.** Turn on the cluster's API server audit logs (control plane logging on EKS, `kube-audit` diagnostic settings on AKS, Cloud Audit Logs on GKE) and alert on reads of these Secrets by anyone other than the Catalyst agent, cert-manager and Sentry. ## Encrypt the Secrets at rest Kubernetes stores Secrets in etcd. Encrypt them with a key in your cloud's key management service, so a copy of etcd or of a backup doesn't reveal the issuer key: - **EKS:** envelope encryption of Kubernetes Secrets with an AWS KMS key you manage. - **AKS:** KMS etcd encryption with a key in Azure Key Vault. - **GKE:** application-layer secrets encryption with a Cloud KMS key. - **Other clusters:** an `EncryptionConfiguration` with a [KMS provider](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/). Back up the cluster with a tool that keeps Secrets encrypted, or leave these namespaces out of backups: cert-manager can issue a new issuer certificate from the root at any time. ## Rotate the issuer cert-manager renews the issuer certificate in your Secret on schedule, but the agent copies it into the Dapr control plane only when the agent starts. After each renewal, restart the agent, well before the old certificate expires: ```bash kubectl -n cra-agent rollout restart deployment agent ``` Then check that Sentry has the new certificate. The two commands print the same serial number when it does: ```bash kubectl -n cert-manager get secret dapr-trust-bundle -o jsonpath='{.data.tls\.crt}' \ | base64 -d | openssl x509 -noout -serial -enddate kubectl -n root-dapr-system get secret dapr-trust-bundle -o jsonpath='{.data.issuer\.crt}' \ | base64 -d | openssl x509 -noout -serial -enddate ``` Workload certificates already issued stay valid until they expire, and Sentry signs new ones with the new key. With `renewBefore: 720h` you have 30 days to restart the agent, so a monthly scheduled restart, or one triggered by the renewal, is enough. Renewing the issuer doesn't change the trust anchor. Changing the root does, and needs a region rejoin through [Diagrid support](mailto:support@diagrid.io).