Skip to main content

Verify Diagrid's artifacts

Everything you install for a BYOC region carries proof of where it came from. You can check that proof yourself, with open-source tools and without asking Diagrid, both before you install and against what is running in your cluster.

From Catalyst 1.137.0, Diagrid's release pipeline attaches these to the artifacts you pull:

ArtifactWhere you get itWhat it carries
Catalyst imagesus-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public, pinned by digest in the chartA signature, SLSA build provenance, a CycloneDX SBOM for each architecture, and vulnerability statements (OpenVEX) where Diagrid has made any
Catalyst Helm chartoci://public.ecr.aws/diagrid/catalystA signature
Diagrid CLI (from 1.113.0)The install scriptA published checksum and a signature, and from 1.114.0 SLSA Build L3 provenance

Every signature is keyless: Diagrid's release workflow signs with a short-lived certificate that names the workflow, and the signature is recorded in Sigstore's public transparency log. There is no Diagrid key to distribute or rotate. What you check is that the certificate names a Diagrid release workflow running on a release tag.

Not covered: third-party images the chart deploys (Envoy, upstream Dapr, the optional OpenTelemetry add-ons) carry whatever their publishers attach, and Diagrid's Dapr build (dapr), diagrid-piko and k8s-tools are not signed yet.

Before you begin​

Set the values every check below uses:

registry=us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public
issuer=https://token.actions.githubusercontent.com
# Diagrid's image release workflows, on a release tag
images_id='^https://github\.com/diagridio/cloudgrid/\.github/workflows/(publish-docker-image|otel-publish-docker-image|catalyst-operator-publish-docker-image)\.yaml@refs/tags/.+/v.+$'

An image built by any other workflow, from a branch or a pull request, or in another repository, fails this identity. An image's certificate names the release candidate tag it was built from, such as catalyst-dataplane/v1.137.0-rc.4: a final release is the same image, by digest, promoted from its last release candidate.

Check the chart before you install it​

version=1.137.0
digest=$(crane digest public.ecr.aws/diagrid/catalyst:${version})

cosign verify --certificate-oidc-issuer "$issuer" \
  --certificate-identity-regexp '^https://github\.com/diagridio/cloudgrid/\.github/workflows/publish-catalyst-chart-release\.yaml@refs/tags/(catalyst-dataplane/)?v.+$' \
  public.ecr.aws/diagrid/catalyst@${digest}

Check the digest rather than the version: a tag can be moved after you check it. Keep the digest, and confirm crane digest still returns it just before you run helm install or helm upgrade with --version ${version}. Automate upgrades gives you each release's digest to compare against.

Check an image​

The chart pins every Diagrid image by digest, and those digests are what you check. To check the images running in a region, list the digests your cluster pulled:

kubectl get pods -n cra-agent \
  -o jsonpath='{range .items[*]}{range .status.containerStatuses[*]}{.imageID}{"\n"}{end}{end}' | sort -u

Then check each Diagrid image's signature and build provenance:

image=${registry}/sidecar@sha256:<digest>

cosign verify --certificate-oidc-issuer "$issuer" --certificate-identity-regexp "$images_id" "$image"

cosign verify-attestation --type slsaprovenance1 \
  --certificate-oidc-issuer "$issuer" --certificate-identity-regexp "$images_id" "$image" \
  | jq '.payload | @base64d | fromjson | .predicate.buildDefinition.externalParameters.workflow'

The provenance records the repository, commit, workflow and run that built the image. The second command prints the repository, the workflow file and the tag it ran on.

Read an image's SBOM​

Each single-architecture image has a CycloneDX SBOM listing every package in it. A multi-architecture digest has none of its own, so look up the digest for the architecture you run first:

amd64=${registry}/sidecar@$(crane digest --platform linux/amd64 ${registry}/sidecar@sha256:<digest>)

cosign verify-attestation --type cyclonedx \
  --certificate-oidc-issuer "$issuer" --certificate-identity-regexp "$images_id" "$amd64" \
  | jq '.payload | @base64d | fromjson | .predicate' > sidecar.cdx.json

trivy sbom sidecar.cdx.json

Scanning the SBOM is fast and needs no access to the image's layers, which suits an inventory or a scheduled scan.

Apply Diagrid's vulnerability statements​

A scanner reports every known vulnerability in a package, whether or not the image can reach the vulnerable code. When Diagrid has established that a vulnerability does not affect an image, for example because the vulnerable function is not compiled into it, the image carries a signed OpenVEX statement saying so. Trivy reads it from the image and leaves those findings out:

trivy image --vex oci "$amd64"

Statements are attached to single-architecture images, like SBOMs, so scan the digest for your architecture. They are stored in cosign's older attestation format, which Trivy reads. To read them yourself with cosign 3, add --new-bundle-format=false, or cosign looks only at the newer format and finds none:

cosign verify-attestation --type openvex --new-bundle-format=false \
  --certificate-oidc-issuer "$issuer" --certificate-identity-regexp "$images_id" "$amd64" \
  | jq '.payload | @base64d | fromjson | .predicate.statements'

Each statement names the vulnerability, the package and the justification. A vulnerability that is not listed is one Diagrid has not ruled out.

Check images as pods start​

An admission controller can check every Diagrid image as its pods are created, such as Sigstore policy-controller or Kyverno. This policy-controller example only warns. Start there, watch what it reports, and enforce once nothing you rely on fails it:

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: diagrid-catalyst-images
spec:
  mode: warn
  images:
    # One glob per signed Diagrid image. The webhook checks images by digest, so @* matches every digest.
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/cra-agent@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-management@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-gateway@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-mcp@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-all@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/identity-injector@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/private-sentry@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/sidecar@*
    - glob: us-central1-docker.pkg.dev/prj-common-p-shared-79896/reg-p-common-docker-public/catalyst-otel-collector@*
  authorities:
    - name: diagrid-release
      signatureFormat: bundle
      keyless:
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subjectRegExp: ^https://github\.com/diagridio/cloudgrid/\.github/workflows/(publish-docker-image|otel-publish-docker-image|catalyst-operator-publish-docker-image)\.yaml@refs/tags/.+/v.+$
      attestations:
        - name: provenance
          predicateType: https://slsa.dev/provenance/v1

Three things to know about how the policy-controller webhook behaves:

  • It refuses an image it cannot resolve to a digest, even in warn mode. Every image in a covered namespace has to be readable from the controller's pod.
  • It rewrites image tags to digests on the workloads it covers. Catalyst's own images are already digests, but a GitOps tool managing tag-referenced workloads in a covered namespace sees the rewrite as drift.
  • It covers only the namespaces you label policy.sigstore.dev/include=true. Label cra-agent first, and your project namespaces last: the Catalyst agent creates and updates the workloads in them, not you.

Check a mirrored copy​

A signature and its attestations are stored beside the image, as separate artifacts that refer to its digest. A copy of the image alone leaves them behind. To mirror Catalyst to your own registry and still verify it, copy with ORAS, which brings them along:

oras cp --recursive ${registry}/sidecar@sha256:<digest> my-registry.example.com/sidecar@sha256:<digest>

The vulnerability statements are stored under a tag named after each digest, sha256-<hex>.att, which --recursive does not follow, so copy that tag for each per-architecture digest too. The mirror script in Private container images does all of this when ORAS and jq are installed. Copy the chart the same way, since helm push gives it a new digest that its signature does not cover.

The digest does not change, so the same commands verify the mirror: run them against my-registry.example.com/<image>@sha256:<digest>, and point the admission policy's globs at your registry.

In a cluster without internet access, check against a local copy of Sigstore's trusted root. On a connected machine, run cosign initialize with TUF_ROOT set to an empty directory, copy the trusted_root.json it writes into your environment, and add --trusted-root trusted_root.json to each cosign verify command. Refresh the copy on a schedule: Sigstore rotates its keys from time to time.

Check the Diagrid CLI​

From CLI 1.113.0, the install script checks every download against its published checksum before installing it, and stops if the checksum is missing or does not match. When cosign is installed, it also checks the download's signature. You do not need to do anything for this; set DIAGRID_SKIP_VERIFY=true only if you have to turn it off.

From CLI 1.114.0, each release also publishes SLSA Build L3 provenance, diagrid.intoto.jsonl, beside its archives. It is signed by the SLSA GitHub generator in a job the build cannot reach. Check a downloaded archive with slsa-verifier:

slsa-verifier verify-artifact diagrid_linux_amd64.tar.gz \
  --provenance-path diagrid.intoto.jsonl \
  --source-uri github.com/diagridio/cloudgrid --source-tag cli/v1.114.0

Managed BYOC and Enterprise Server​

In Managed BYOC, Diagrid installs and upgrades the region, but it runs in your account and you can check it the same way: list the digests your cluster pulled, as in Check an image, and verify them. Talk to Diagrid before you add an admission policy to a managed region's cluster, so that a refused image cannot stall an upgrade.

In Enterprise Server, the control plane images Diagrid delivers for your registry are signed too, from control plane 1.186.0. Their web console image is built by one more release workflow, so add admingrid-release to the list in images_id. Mirror them with ORAS, as above, to keep their signatures. The control plane's Helm charts are not signed yet.

Next steps​

Last updated on