Sign every image, then refuse the ones that are not signed

This is a sample post, here so the layout can be checked. Delete content/posts/sample-signing-every-image.md and its cover when you are done with it.

Most supply-chain work stalls at the same place. The pipeline signs images, the dashboard turns green, and nothing downstream ever checks the signature. Signing on its own is a claim. It only becomes a control when something refuses to run what it cannot verify.

Signing is the easy half

Adding a signing step costs an afternoon. Cosign reads the digest the build just produced and signs it keylessly against the workflow identity:

# Sign by digest, never by tag: a tag can be repointed after signing.
DIGEST=$(crane digest "$IMAGE:$TAG")
cosign sign --yes "$IMAGE@$DIGEST"

cosign attest --yes \
  --predicate sbom.spdx.json \
  --type spdxjson \
  "$IMAGE@$DIGEST"

Signing by digest matters more than it looks. A tag is a mutable pointer: sign app:v1.2.0 and someone can push a different image to that tag an hour later, and the signature still appears to cover it.

What the attestation is for

A signature says the image came from somewhere you trust. An attestation says what is in it. The SBOM is the part that answers the question you will actually be asked, usually at speed, usually about a CVE announced that morning: which of our running images contain this package?

The half that changes anything

The control is admission. Until the cluster refuses an unsigned image, signing has changed nothing about what can run:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - 'registry.example.com/platform/*'
          attestors:
            - entries:
                - keyless:
                    issuer: https://token.actions.githubusercontent.com
                    subject: https://github.com/acme/platform/*

Note validationFailureAction: Enforce. In Audit this policy produces a report nobody reads.

Rolling it out without stopping everyone

Going straight to Enforce across every namespace breaks the first deploy that was not in the pilot, usually someone else's, usually on a Friday. What has worked for me:

1. Run in Audit for a fortnight and read the report daily. The point is to find the images nobody remembered: the vendor sidecar, the ancient init container, the one built on a laptop in 2021.
2. Enforce in one namespace that a team owns end to end, with that team expecting it.
3. Widen by namespace, not by percentage. A namespace has an owner who can be told; a percentage does not.
4. Keep a documented break-glass label, time-boxed and alerting when used.

The policy that gets rolled back after one bad week protects nothing. The boring staged version survives, and surviving is the whole point.

What to watch afterwards

Rejections are the signal. A rejection means something tried to run code of unknown origin, which is worth a look even when it turns out to be a vendor image that was never going to be signed. Alert on the rate, not each event: one rejection is a misconfigured deploy, fifty is a pipeline that stopped signing.