# Verify Diagrid's artifacts

Everything you install for a BYOC region carries proof of where it came from. You can check that proof yourself, with open-source tools and without asking Diagrid, both before you install and against what is running in your cluster.

From Catalyst **1.137.0**, Diagrid's release pipeline attaches these to the artifacts you pull:

| Artifact | Where you get it | What it carries |
|---|---|---|
| Catalyst images | `us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public`, pinned by digest in the chart | A signature, SLSA build provenance, a CycloneDX SBOM for each architecture, and vulnerability statements (OpenVEX) where Diagrid has made any |
| Catalyst Helm chart | `oci://public.ecr.aws/diagrid/catalyst` | A signature |
| Diagrid CLI (from 1.113.0) | The [install script](https://docs.diagrid.io/getting-started/local-installation#diagrid-cli) | A published checksum and a signature, and from 1.114.0 SLSA Build L3 provenance |

Every signature is keyless: Diagrid's release workflow signs with a short-lived certificate that names the workflow, and the signature is recorded in [Sigstore's](https://www.sigstore.dev/) public transparency log. There is no Diagrid key to distribute or rotate. What you check is that the certificate names a Diagrid release workflow running on a release tag.

Not covered: third-party images the chart deploys (Envoy, upstream Dapr, the optional OpenTelemetry add-ons) carry whatever their publishers attach, and Diagrid's Dapr build (`dapr`), `diagrid-piko` and `k8s-tools` are not signed yet.

## Before you begin

- [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) 3 or later, to check signatures and attestations.
- [crane](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md), to look up digests.
- Optionally [slsa-verifier](https://github.com/slsa-framework/slsa-verifier), [Trivy](https://trivy.dev/) and [ORAS](https://oras.land/), for the sections that use them.

Set the values every check below uses:

```bash
registry=us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public
issuer=https://token.actions.githubusercontent.com
# Diagrid's image release workflows, on a release tag
images_id='^https://github\.com/diagridio/cloudgrid/\.github/workflows/(publish-docker-image|otel-publish-docker-image|catalyst-operator-publish-docker-image)\.yaml@refs/tags/.+/v.+$'
```

An image built by any other workflow, from a branch or a pull request, or in another repository, fails this identity. An image's certificate names the release candidate tag it was built from, such as `catalyst-dataplane/v1.137.0-rc.4`: a final release is the same image, by digest, promoted from its last release candidate.

## Check the chart before you install it

```bash
version=1.137.0
digest=$(crane digest public.ecr.aws/diagrid/catalyst:${version})

cosign verify --certificate-oidc-issuer "$issuer" \
  --certificate-identity-regexp '^https://github\.com/diagridio/cloudgrid/\.github/workflows/publish-catalyst-chart-release\.yaml@refs/tags/(catalyst-dataplane/)?v.+$' \
  public.ecr.aws/diagrid/catalyst@${digest}
```

Check the digest rather than the version: a tag can be moved after you check it. Keep the digest, and confirm `crane digest` still returns it just before you run `helm install` or `helm upgrade` with `--version ${version}`. [Automate upgrades](./automate-upgrades.mdx) gives you each release's digest to compare against.

## Check an image

The chart pins every Diagrid image by digest, and those digests are what you check. To check the images running in a region, list the digests your cluster pulled:

```bash
kubectl get pods -n cra-agent \
  -o jsonpath='{range .items[*]}{range .status.containerStatuses[*]}{.imageID}{"\n"}{end}{end}' | sort -u
```

Then check each Diagrid image's signature and build provenance:

```bash
image=${registry}/sidecar@sha256:<digest>

cosign verify --certificate-oidc-issuer "$issuer" --certificate-identity-regexp "$images_id" "$image"

cosign verify-attestation --type slsaprovenance1 \
  --certificate-oidc-issuer "$issuer" --certificate-identity-regexp "$images_id" "$image" \
  | jq '.payload | @base64d | fromjson | .predicate.buildDefinition.externalParameters.workflow'
```

The provenance records the repository, commit, workflow and run that built the image. The second command prints the repository, the workflow file and the tag it ran on.

## Read an image's SBOM

Each single-architecture image has a CycloneDX SBOM listing every package in it. A multi-architecture digest has none of its own, so look up the digest for the architecture you run first:

```bash
amd64=${registry}/sidecar@$(crane digest --platform linux/amd64 ${registry}/sidecar@sha256:<digest>)

cosign verify-attestation --type cyclonedx \
  --certificate-oidc-issuer "$issuer" --certificate-identity-regexp "$images_id" "$amd64" \
  | jq '.payload | @base64d | fromjson | .predicate' > sidecar.cdx.json

trivy sbom sidecar.cdx.json
```

Scanning the SBOM is fast and needs no access to the image's layers, which suits an inventory or a scheduled scan.

## Apply Diagrid's vulnerability statements

A scanner reports every known vulnerability in a package, whether or not the image can reach the vulnerable code. When Diagrid has established that a vulnerability does not affect an image, for example because the vulnerable function is not compiled into it, the image carries a signed OpenVEX statement saying so. Trivy reads it from the image and leaves those findings out:

```bash
trivy image --vex oci "$amd64"
```

Statements are attached to single-architecture images, like SBOMs, so scan the digest for your architecture. They are stored in cosign's older attestation format, which Trivy reads. To read them yourself with cosign 3, add `--new-bundle-format=false`, or cosign looks only at the newer format and finds none:

```bash
cosign verify-attestation --type openvex --new-bundle-format=false \
  --certificate-oidc-issuer "$issuer" --certificate-identity-regexp "$images_id" "$amd64" \
  | jq '.payload | @base64d | fromjson | .predicate.statements'
```

Each statement names the vulnerability, the package and the justification. A vulnerability that is not listed is one Diagrid has not ruled out.

## Check images as pods start

An admission controller can check every Diagrid image as its pods are created, such as [Sigstore policy-controller](https://docs.sigstore.dev/policy-controller/overview/) or [Kyverno](https://kyverno.io/docs/policy-types/cluster-policy/verify-images/). This policy-controller example only warns. Start there, watch what it reports, and enforce once nothing you rely on fails it:

```yaml
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: diagrid-catalyst-images
spec:
  mode: warn
  images:
    # One glob per signed Diagrid image. The webhook checks images by digest, so @* matches every digest.
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/cra-agent@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-management@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-gateway@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-mcp@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-all@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/identity-injector@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/private-sentry@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/sidecar@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-otel-collector@*
  authorities:
    - name: diagrid-release
      signatureFormat: bundle
      keyless:
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subjectRegExp: ^https://github\.com/diagridio/cloudgrid/\.github/workflows/(publish-docker-image|otel-publish-docker-image|catalyst-operator-publish-docker-image)\.yaml@refs/tags/.+/v.+$
      attestations:
        - name: provenance
          predicateType: https://slsa.dev/provenance/v1
```

Three things to know about how the policy-controller webhook behaves:

- **It refuses an image it cannot resolve to a digest, even in warn mode.** Every image in a covered namespace has to be readable from the controller's pod.
- **It rewrites image tags to digests** on the workloads it covers. Catalyst's own images are already digests, but a GitOps tool managing tag-referenced workloads in a covered namespace sees the rewrite as drift.
- **It covers only the namespaces you label** `policy.sigstore.dev/include=true`. Label `cra-agent` first, and your project namespaces last: the Catalyst agent creates and updates the workloads in them, not you.

## Check a mirrored copy

A signature and its attestations are stored beside the image, as separate artifacts that refer to its digest. A copy of the image alone leaves them behind. To mirror Catalyst to your own registry and still verify it, copy with [ORAS](https://oras.land/docs/commands/oras_copy), which brings them along:

```bash
oras cp --recursive ${registry}/sidecar@sha256:<digest> my-registry.example.com/sidecar@sha256:<digest>
```

The vulnerability statements are stored under a tag named after each digest, `sha256-<hex>.att`, which `--recursive` does not follow, so copy that tag for each per-architecture digest too. The mirror script in [Private container images](./production-planning.mdx#private-container-images) does all of this when ORAS and jq are installed. Copy the chart the same way, since `helm push` gives it a new digest that its signature does not cover.

The digest does not change, so the same commands verify the mirror: run them against `my-registry.example.com/<image>@sha256:<digest>`, and point the admission policy's globs at your registry.

In a cluster without internet access, check against a local copy of Sigstore's trusted root. On a connected machine, run `cosign initialize` with `TUF_ROOT` set to an empty directory, copy the `trusted_root.json` it writes into your environment, and add `--trusted-root trusted_root.json` to each `cosign verify` command. Refresh the copy on a schedule: Sigstore rotates its keys from time to time.

## Check the Diagrid CLI

From CLI 1.113.0, the install script checks every download against its published checksum before installing it, and stops if the checksum is missing or does not match. When cosign is installed, it also checks the download's signature. You do not need to do anything for this; set `DIAGRID_SKIP_VERIFY=true` only if you have to turn it off.

From CLI 1.114.0, each release also publishes SLSA Build L3 provenance, `diagrid.intoto.jsonl`, beside its archives. It is signed by the [SLSA GitHub generator](https://github.com/slsa-framework/slsa-github-generator) in a job the build cannot reach. Check a downloaded archive with slsa-verifier:

```bash
slsa-verifier verify-artifact diagrid_linux_amd64.tar.gz \
  --provenance-path diagrid.intoto.jsonl \
  --source-uri github.com/diagridio/cloudgrid --source-tag cli/v1.114.0
```

## Managed BYOC and Enterprise Server

In [Managed BYOC](https://docs.diagrid.io/operate/hosting/byoc/managed), Diagrid installs and upgrades the region, but it runs in your account and you can check it the same way: list the digests your cluster pulled, as in [Check an image](#check-an-image), and verify them. Talk to Diagrid before you add an admission policy to a managed region's cluster, so that a refused image cannot stall an upgrade.

In [Enterprise Server](https://docs.diagrid.io/operate/hosting/enterprise-server), the control plane images Diagrid delivers for your registry are signed too, from control plane 1.186.0. Their web console image is built by one more release workflow, so add `admingrid-release` to the list in `images_id`. Mirror them with ORAS, as above, to keep their signatures. The control plane's Helm charts are not signed yet.

## Next steps

- [Private container images](./production-planning.mdx#private-container-images): mirror Catalyst to your own registry.
- [Lifecycle management](./upgrades-operations.md): check each new version before you upgrade to it.