Recommendations

This document lists recommendations from the Security SIG for the consideration of OpenTelemetry SIGs.

Scanning

CodeQL scanning

The organization uses CodeQL for semantic analysis of the code in various repositories. The recommendation is to run CodeQL on every pull request and on every commit to the main branch.

Issue: (#15)

zizmor scanning

zizmor is a static analysis tool for GitHub Actions workflows. The recommendation is to run zizmor on every pull request and on every commit to the main branch for repositories using GitHub Actions.

GitHub Actions workflows are part of the project supply chain. zizmor can help identify CI/CD security issues such as template injection, unsafe credential handling, overly broad token permissions, risky workflow triggers, and ambiguous or mutable action references.

Where possible, repositories should make zizmor a required code scanning check by default, with documented opt-out handling for repositories where it is not yet applicable.

Resources:

Issue: (#268)

Integrity

Sign release artifacts with Sigstore Cosign

Checksums help detect corruption, but they do not authenticate who produced an artifact when the artifact and checksum are downloaded from the same location. If both files are replaced, checksum verification can still succeed for malicious content.

Use Sigstore Cosign to sign release container images and other consumer-facing artifacts, such as binaries, archives, SBOMs, and checksum manifests. Publish instructions that let consumers verify both the artifact and the expected signer identity.

Use Cosign's keyless signing in automated release workflows. With keyless signing, the workflow uses an OpenID Connect (OIDC) identity and a short-lived certificate instead of a long-lived signing key. The signing event is recorded in Sigstore's transparency infrastructure and can be audited.

Sign in GitHub Actions

Use a dedicated signing job that runs only for trusted release events, such as version tags. Grant id-token: write only to the job that needs the GitHub OIDC token, and grant no other permissions beyond those needed to read or publish the artifacts.

The following excerpt installs Cosign and signs a container image after it has been pushed. Sign the immutable image digest, not only a mutable tag.

jobs:
  sign:
    if: startsWith(github.ref, 'refs/tags/')
    permissions:
      contents: read
      id-token: write
      packages: write # Include only when required by the target registry.
    steps:
      - name: Install Cosign
        uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2

      # Run after the image push step, whose output is the manifest digest.
      - name: Sign container image
        env:
          IMAGE: ghcr.io/open-telemetry/example@${{ steps.build.outputs.digest }}
        run: cosign sign --yes "${IMAGE}"

For downloadable files, finalize the files before signing them and use a Sigstore bundle rather than separate signature and certificate files. Generate checksum manifests before signing, and do not add generated bundles to the checksum manifest or sign the bundles themselves.

artifact=dist/example-v1.2.3-linux-amd64.tar.gz

cosign sign-blob --yes \
  --bundle "${artifact}.sigstore.json" \
  "${artifact}"

Publish each bundle next to its artifact. Do not modify an artifact after it is signed. Before publishing a release, verify the image signatures and bundles in CI using the same certificate identity and OIDC issuer that users will rely on.

Document verification

Document copy-pasteable verification commands for every class of signed artifact. Bind verification to the exact release workflow and tag using --certificate-identity; a repository-wide identity regular expression could accept a signature from an unrelated workflow in the same repository.

For example:

repository=open-telemetry/example
release_tag=v1.2.3
certificate_identity="https://github.com/${repository}/.github/workflows/release.yml@refs/tags/${release_tag}"
certificate_oidc_issuer=https://token.actions.githubusercontent.com
image="ghcr.io/open-telemetry/example@sha256:IMAGE_DIGEST"

cosign verify \
  --certificate-identity "${certificate_identity}" \
  --certificate-oidc-issuer "${certificate_oidc_issuer}" \
  "${image}"

artifact=example-v1.2.3-linux-amd64.tar.gz
cosign verify-blob "${artifact}" \
  --bundle "${artifact}.sigstore.json" \
  --certificate-identity "${certificate_identity}" \
  --certificate-oidc-issuer "${certificate_oidc_issuer}"

If verification fails, consumers should stop and must not run, deploy, or compile the artifact.

Signatures authenticate the final bytes and signer identity, but do not by themselves describe how the artifact was built. Checksums, SBOMs, and build provenance attestations provide complementary information.

Resources:

Attest release artifacts with GitHub Artifact Attestations

Build provenance attestations help consumers trace a release artifact to the repository, commit, workflow, and CI environment that produced it. This gives auditors and automated policy checks evidence about where and how an artifact was built, while binding that evidence to the artifact's digest.

Use GitHub Artifact Attestations for consumer-facing artifacts built in GitHub Actions, such as binaries, archives, packages, software bills of materials (SBOMs), checksum manifests, and container images. The actions/attest action generates a signed in-toto statement with SLSA build provenance, using the workflow's GitHub OIDC identity and a short-lived Sigstore certificate.

Attestations complement artifact signatures. A signature authenticates the finished bytes and signer identity; a provenance attestation adds signed claims about the build. Neither proves that the source or artifact is secure, so consumers must verify the attestation against an expected identity and policy.

Generate build provenance in GitHub Actions

Generate attestations only for trusted release events, such as version tags. Finalize each artifact before attesting it, because the attestation records the digest of the exact bytes present at that point. Do not modify the artifact afterward.

Grant id-token: write and attestations: write only to the job that creates the attestation. Grant no other permissions beyond those the build and publication steps require. Pin the action to a full commit SHA.

The following excerpt attests a downloadable release archive after it has been built and packaged:

jobs:
  release:
    if: startsWith(github.ref, 'refs/tags/')
    permissions:
      attestations: write
      contents: read
      id-token: write
    steps:
      # Build and finalize the release archive before this step.
      - name: Attest release artifact
        uses: actions/attest@f7c74d28b9d84cb8768d0b8ca14a4bac6ef463e6 # v4.2.0
        with:
          subject-path: dist/example-v1.2.3-linux-amd64.tar.gz

Use a multiline path or glob to attest multiple finalized files.

Container image attestations

Push a container image before attesting it. Pass its fully qualified name without a tag as subject-name, pass the immutable digest from the build step as subject-digest, and set push-to-registry: true so the signed bundle is stored with the image in the OCI registry.

The following excerpt assumes that the source has been checked out and the workflow has authenticated to GitHub Container Registry (GHCR):

jobs:
  release-container:
    if: startsWith(github.ref, 'refs/tags/')
    permissions:
      artifact-metadata: write # Create a linked artifact storage record.
      attestations: write
      contents: read
      id-token: write
      packages: write # Publish the image and attestation to GHCR.
    env:
      IMAGE: ghcr.io/open-telemetry/example
    steps:
      - name: Build and push container image
        id: build-image
        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
        with:
          context: .
          push: true
          tags: ${{ env.IMAGE }}:${{ github.ref_name }}
          provenance: true
          sbom: true

      - name: Attest container image
        uses: actions/attest@f7c74d28b9d84cb8768d0b8ca14a4bac6ef463e6 # v4.2.0
        with:
          subject-name: ${{ env.IMAGE }}
          subject-digest: ${{ steps.build-image.outputs.digest }}
          push-to-registry: true

The provenance and sbom inputs attach Docker BuildKit provenance and SBOM metadata to the image. The actions/attest step separately creates a Sigstore-signed GitHub build provenance attestation for the pushed image digest. These attestations are complementary.

Use packages: write only when required to publish to the target registry. The actions/attest action creates a linked artifact storage record by default; set create-storage-record: false and omit artifact-metadata: write when that record is not needed. Use the same fully qualified registry hostname throughout the workflow and verification instructions. For Docker Hub, use docker.io as the registry portion of subject-name.

Verify and enforce provenance

Generating attestations alone provides no security benefit. Publish copy-pasteable verification commands and enforce them in artifact promotion, deployment, or admission workflows.

Verify the repository, the exact signer workflow, and the expected release ref. Using only --owner trusts attestations from every repository owned by that organization and is usually too broad.

For example:

repository=open-telemetry/example
release_tag=v1.2.3
signer_workflow="${repository}/.github/workflows/release.yml"
artifact=example-v1.2.3-linux-amd64.tar.gz

gh attestation verify "${artifact}" \
  --repo "${repository}" \
  --signer-workflow "${signer_workflow}" \
  --source-ref "refs/tags/${release_tag}"

image=oci://ghcr.io/open-telemetry/example@sha256:IMAGE_DIGEST

# Fetch the attestation from the GitHub API.
gh attestation verify "${image}" \
  --repo "${repository}" \
  --signer-workflow "${signer_workflow}" \
  --source-ref "refs/tags/${release_tag}"

# Fetch the attestation from the OCI registry.
gh attestation verify "${image}" \
  --bundle-from-oci \
  --repo "${repository}" \
  --signer-workflow "${signer_workflow}" \
  --source-ref "refs/tags/${release_tag}"

Each container image command independently verifies the signature, image digest, repository, signer workflow, and source ref. Run both during release validation to confirm that the attestation is available through both GitHub and the OCI registry. Consumers need only one successful verification unless their policy requires both sources; running both confirms publication, but does not add cryptographic assurance.

Without --bundle-from-oci, gh reads the signed bundle from the GitHub API. The flag instead reads the bundle stored alongside the image in the OCI registry. Authenticate to a private registry before either verification. Append --format json to inspect the verified GitHub/Sigstore bundle, including its certificate and signed SLSA provenance statement. Neither command inspects the Docker BuildKit provenance or SBOM attestations.

If the build uses a reusable workflow, verify the reusable workflow's identity as the signer. If verification fails, consumers should stop and must not run, deploy, or compile the artifact.

A compromised build workflow or runner can still produce malicious artifacts and attest them. Protect release workflows, use isolated runners, minimize job permissions, and consider a vetted reusable workflow as a trusted builder when stronger isolation is required.

Resources:

GitHub immutable releases

GitHub supports immutable releases, which prevent release assets from being modified after a release is published. This helps reduce supply-chain risk by making it harder for an attacker (or a compromised account/token) to silently swap binaries, SBOMs, or other artifacts after consumers have started downloading or verifying them.

Recommendation: enable immutable releases and treat each published release as a permanent, verifiable record.

Best practices:

Consider creating releases as drafts first. This allows you to attach all assets before the release becomes immutable.
Prefer a fully automated, reproducible release process (CI builds artifacts from a tagged commit, then publishes the release). Avoid building artifacts on developer workstations.
Publish integrity metadata alongside assets (for example: checksums, SBOMs, provenance/attestations, and/or signatures) so downstream users can verify what they downloaded matches what you produced.
Restrict who/what can publish releases:
- Use the smallest possible GitHub Actions permissions for release workflows.
If you need to fix a bad release, publish a new release (and clearly mark the old one as deprecated) rather than replacing assets in-place.

Resources:

GitHub environment secrets

GitHub environment secrets provide an additional layer of protection for sensitive secrets used in publishing, signing, and other privileged workflows.

With repository secrets, any workflow running in the repository can access them, including workflows triggered on non-protected branches. This means anyone with Write access could push a non-protected branch containing secret exfiltration code and trigger a workflow without going through a PR review.

With environment secrets, access is restricted to workflows running in the context of a named environment, and that environment can be configured to only allow deployments from specific branches. This means even a contributor with Write access cannot access the secrets without their code successfully passing all branch protection criteria (i.e. an approved and merged PR).

Recommendation: migrate publishing, signing, and other privileged secrets from repository secrets to an environment with a deployment branch policy restricting access to main and release/** branches.

Steps:

Create and configure an environment (e.g., protected) with a deployment branch policy via Terraform in open-telemetry/admin, allowing only main and release/** (adjust to match your branching strategy).
Request admin permission to manage secrets for the environment. See Request Repository Admin Permissions.
Add your publishing and signing secrets to the environment.
Update release workflows to run in the context of the environment:
```
jobs:
  release:
    environment: protected
    steps:
      ...
```
Note: if you ever need to make an older patch release from a release branch, backport this workflow change to that branch first.
Remove the corresponding repository-level secrets. If both exist, the repository-level secret remains accessible from any branch, defeating the purpose.

Resources:

Binding to Network Interfaces

Always bind to localhost rather than to 0.0.0.0 or any interface, unless there is a specific need to do otherwise. Binding to localhost reduces the attack surface of your application by making it only accessible to devices on the same machine. Binding to 0.0.0.0 or "all" interfaces can make your application respond on all current and future network interfaces.

Issue: (#19)

Container images

Use the smallest image possible

While the size of an image doesn't matter for the runtime, we are responsible for everything that is in the image. If you are delivering a statically linked binary, you can probably build a scratch image for production purposes. It might be a good idea to provide a second set of images for debugging purposes, such as ones including a shell and networking utilities.

If you need to include files in the image, such as root certificates, use the builder pattern to obtain the files and copy only those into the final image. Example:

FROM alpine:3.20 as certificates
RUN apk --no-cache add ca-certificates

FROM scratch
COPY --from=certificates /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY mybinary mybinary

Use a non-privileged user

Under some circumstances, your image might be executed as a privileged user by default. In others, like a hardened Kubernetes distribution, your container is forced to run using a non-privileged account and bugs might appear as the image wasn't built with that in mind. It's recommended to set a high user ID when building the image, to ensure the process always runs as a non-privileged user. This will not only make it more secure by default but will help you uncover issues before they affect security-conscious users.

Here's how to do it:

USER 65532:65532

Keep your images up to date

Use tools like Renovate or Dependabot to achieve that.

Renovate

Here's an example of a PR created by Renovate on a repository using its default configuration.

If you are interested in using Renovate, open an issue requesting the TC to enable it to your repository, like this. You should then receive a PR from Renovate doing the onboarding, like this.

Dependabot

For directories containing a Dockerfile, add the following to your .github/dependabot.yml:

version: 2
updates:
  - package-ecosystem: docker
    directory: /
    schedule:
      interval: weekly

And here's one from Dependabot.

Navigation Menu

Uh oh!

Recommendations

Scanning

CodeQL scanning

zizmor scanning

Integrity

Sign release artifacts with Sigstore Cosign

Sign in GitHub Actions

Document verification

Attest release artifacts with GitHub Artifact Attestations

Generate build provenance in GitHub Actions

Container image attestations

Verify and enforce provenance

GitHub immutable releases

GitHub environment secrets

Binding to Network Interfaces

Container images

Use the smallest image possible

Use a non-privileged user

Keep your images up to date

Renovate

Dependabot

Uh oh!

FilesExpand file tree

recommendations.md

Latest commit

History

recommendations.md

File metadata and controls

Recommendations

Scanning

CodeQL scanning

zizmor scanning

Integrity

Sign release artifacts with Sigstore Cosign

Sign in GitHub Actions

Document verification

Attest release artifacts with GitHub Artifact Attestations

Generate build provenance in GitHub Actions

Container image attestations

Verify and enforce provenance

GitHub immutable releases

GitHub environment secrets

Binding to Network Interfaces

Container images

Use the smallest image possible

Use a non-privileged user

Keep your images up to date

Renovate

Dependabot