Signed, Sealed, Delivered - Sign and Verify All the Things - Jeremy Rickard, Microsoft

Signed, Sealed, Delivered - Sign and Verify All the Things

Speakers: Jeremy Rickard, Principal Software Engineer, Microsoft Azure, Co-chair of SIG Release in the Kubernetes project

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=g--50XLcqRw

Overview

In the rapidly evolving landscape of cloud-native development, securing the software supply chain has become a paramount concern. Jeremy Rickard, a Principal Software Engineer at Microsoft Azure and co-chair of SIG Release in the Kubernetes project, delivered a compelling talk at KubeCon EU addressing this challenge. His presentation, "Signed, Sealed, Delivered - Sign and Verify All the Things," delved into the critical importance of signing and verifying OCI artifacts to ensure the authenticity and integrity of resources deployed within Kubernetes environments.

Rickard's talk highlighted the dramatic shift from the early, "carefree" days of container deployment to the current reality fraught with vulnerabilities, rate limits, and the constant threat of malicious injection. He demonstrated a practical, end-to-end solution for establishing a robust container supply chain security posture, leveraging a suite of entirely CNCF ecosystem projects. The core message emphasizes that by taking control of the software supply chain through mirroring, scanning, signing, and rigorous verification, organizations can mitigate significant risks and build a more secure foundation for their cloud-native applications.

Background

▶ Watch: Introduction: The journey from carefree to complex container security (0:00)

The journey into containerization, for many, began around 2017, characterized by a seemingly effortless ability to package applications and deploy them with a simple docker push. DockerHub served as a ubiquitous, free registry, fostering a "carefree" development experience where deploying to production felt straightforward. However, this simplicity has given way to a complex reality marked by significant security and operational challenges.

Today's landscape is far from carefree. Organizations routinely encounter DockerHub rate limits, which can disrupt scale operations or new application spin-ups, especially for free accounts. The fragility of relying on external registries was starkly illustrated by the disruptive migration for Kubernetes users from GCR to registry.k8s.io, highlighting the risk of losing control over critical dependencies when registries change or disappear. Beyond operational hurdles, the most pressing concern is the pervasive issue of vulnerabilities in containers. Rickard cited an example from the demo application, the "emoji vote app," which in its original form contained 394 high and 61 critical vulnerabilities, a common scenario for many organizations battling regulatory compliance and daily remediation efforts. Relying on third-party registries often means dependency on upstream projects or vendors for fixes, a situation that can be slow and reactive.

A more insidious threat involves registry compromises, where malicious actors can push compromised versions of popular images (e.g., Kong) containing crypto miners or other nefarious payloads. This risk extends beyond container images to deployment manifests and other resources, where changes in a Helm chart could introduce bad defaults or expose security vulnerabilities. The talk underscores that the scope of supply chain security now encompasses all OCI artifacts—anything storable in a registry, not just containers. This includes Helm charts, Flux configurations, SBOMs (Software Bill of Materials), and vulnerability reports, all of which benefit from the same distribution mechanisms, immutable references, and centralized authentication/authorization schemes offered by OCI-compatible registries.

Key Findings

▶ Watch: Threats beyond images: compromised registries and deployment manifests (4:00)

Jeremy Rickard's presentation delivers several critical findings, demonstrating that a comprehensive and robust container supply chain security strategy is not only necessary but also entirely achievable using existing open-source tools within the CNCF ecosystem.

Firstly, the talk conclusively shows that it is possible to establish an end-to-end secure software supply chain for OCI artifacts by adopting a multi-layered approach. This involves mirroring external dependencies to an internal, controlled registry, followed by rigorous scanning for vulnerabilities, proactive patching where necessary, and crucially, signing all artifacts. This process ensures that organizations maintain full control over their dependencies and have a verifiable chain of custody.

Secondly, the presentation highlights the effectiveness of verification at deployment time. By enforcing policies within the Kubernetes cluster that only permit artifacts from trusted internal registries and require valid signatures, organizations can prevent unauthorized or tampered resources from ever running in production. This shifts security left, catching issues before they become runtime problems.

Thirdly, a significant finding is the maturity and interoperability of CNCF projects for this purpose. Rickard successfully demonstrates a stack composed entirely of CNCF tools—Zot for the registry, Flux for GitOps deployments, Notation for signing, Copa for patching, ORAS for registry client operations, and Kyverno for policy enforcement—all working cohesively to secure the supply chain. This underscores that organizations do not necessarily need commercial solutions for foundational supply chain security.

Finally, the concept of a "chain of custody" for digital artifacts, analogous to the physical world of mail-in ballots, is a core finding. Just as a ballot's signature provides an attestation of its origin, digital signatures on OCI artifacts provide verifiable proof of who produced or approved an artifact at each stage of its lifecycle. This allows for clear attestation and rejection of anything that does not meet defined trust policies, thereby bolstering the authenticity and integrity of deployed applications.

Technical Deep Dive

▶ Watch: Practical demo: Signing container images with Cosign (9:30)

The technical core of Rickard's talk revolves around transforming the "angry experience" of container deployment into a secure, controlled process. This is achieved through a systematic application of well-known practices across the OCI artifact lifecycle, all powered by CNCF projects.

The foundation of this approach is the recognition that modern cloud-native supply chains involve more than just container images. The OCI spec has evolved to accommodate various artifact types—such as Helm charts, Flux configurations, SBOMs, and vulnerability reports—within registries. Tools like ORAS (OCI Registry As Storage) are instrumental in pushing and pulling these diverse artifacts, leveraging the same distribution and immutable referencing mechanisms as container images. This consolidation simplifies management, authentication, and authorization.

The process begins with mirroring all external dependencies (e.g., from DockerHub or GCR) into an internal registry. This provides several benefits: mitigating rate limits, centralizing control, and creating a single source of truth for all consumed artifacts. Once mirrored, these images undergo a critical security enhancement phase:

1. Scanning: Images are scanned for vulnerabilities using tools like Trivy (from Aqua Security).
2. Patching: If vulnerabilities are found, projects like Copa can automatically generate patch layers to remediate known issues, creating a "cleaner" image.
3. Signing: After scanning and potential patching, the images are signed. This is a crucial step that provides attestation—a cryptographic guarantee that the image was produced by a trusted entity and has not been tampered with since. The talk highlights Notation as the signing tool, which generates a signature object stored alongside the artifact in the registry. This signature object contains metadata about the container and is discoverable by verification tooling.

Once artifacts are in the internal registry and signed, the next critical phase is verification on deployment. This ensures that only authorized and untampered artifacts are allowed to run in the cluster. Tools like Kyverno (or Gatekeeper) are used as admission controllers in Kubernetes to enforce these policies. Kyverno can be configured to:

• Restrict image sources: Ensuring all images originate from the approved internal registry (e.g., zot.jeremyrickard.com).
• Verify signatures: Utilizing Notation's built-in verifier, Kyverno checks the signature of the deploying artifact (e.g., a pod's container image) against a predefined trust policy. This policy specifies the trusted certificate(s) and the scope within the registry (e.g., zot.jeremyrickard.com/*). If the signature does not match or is missing, the deployment is rejected.

The speaker's chosen CNCF stack for the demo illustrates this integration:

• Zot: A lightweight OCI-compatible registry, serving as the central repository for all mirrored, scanned, patched, and signed artifacts.
• Flux: A GitOps continuous delivery solution, responsible for pulling deployment manifests and applying them to the cluster. Flux's support for OCI-based artifacts and Notation verification is key here.
• Notation: The signing tool, used to create and verify digital signatures for OCI artifacts. It supports plugins for secure key management, such as Azure Key Vault (AKV) used in the demo.
• Copa: A container patching tool that automates the creation of new container layers to fix vulnerabilities identified by scanners like Trivy.
• ORAS: The command-line utility for interacting with OCI registries, used for copying and managing artifacts.
• Kyverno: A Kubernetes native policy engine that enforces security policies, including image source restrictions and signature verification, at admission time.

This integrated approach provides a robust framework for managing the authenticity and integrity of all resources throughout the software supply chain, from development to deployment.

Demo / Proof of Concept

▶ Watch: Practical demo: Verifying signed images using Cosign (12:00)

The live demonstration was a cornerstone of the talk, showcasing a complete, end-to-end secure supply chain for a Kubernetes application using solely CNCF projects. The speaker used the "emoji vote app" from 2017, intentionally selecting an old, vulnerable application to highlight the improvements possible with modern security practices.

The demo began with the setup of a Kubernetes cluster (AKS) and an internal Zot registry. The core of the automated process was a GitHub Actions pipeline designed to:

1. Retag and Copy Images: The pipeline identified all container images required by the emoji vote app, Flux, and Kyverno (originally from DockerHub or GCR). Using ORAS, these images were copied and retagged into the internal zot.jeremyrickard.com registry. This step addressed issues like DockerHub rate limits and dependency on external sources.
2. Sign Images: Each copied image was then signed using Notation. The signing process utilized a certificate stored in Azure Key Vault (for the jereard.com domain), with the AKV plugin for Notation. This generated a verifiable signature object alongside each image in Zot.
3. Scan and Patch: After signing, Trivy (Aqua Security's vulnerability scanner) was used to scan the images. If vulnerabilities were detected, Copa was invoked to automatically generate and apply a patch layer, creating a remediated version of the image, which was then pushed to the registry.

Once the images were secured in the internal registry, the cluster was configured:

1. Flux Installation: Flux, the GitOps operator, was installed into the AKS cluster. Crucially, its own images were pulled from the internal Zot registry, demonstrating the self-application of the security principles.
2. Kyverno Installation and Policy Enforcement: Kyverno, the policy engine, was also installed, with its images sourced from Zot. Two key Kyverno cluster policies were then applied:

• One policy restricted all incoming images to originate only from zot.jeremyrickard.com.
• A second, more critical policy, configured Kyverno's built-in Notation verifier to check for signatures from the speaker's trusted certificate (jereard.com) on all Pod resources. If an image lacked a valid signature from this trusted source, its deployment would be rejected. The speaker initially set this to warn mode to avoid blocking AKS's own system images for the demo, but emphasized it could be configured for strict deny.

Finally, the emoji vote application's deployment manifests were themselves treated as OCI artifacts:

1. Publishing Flux Configuration: The application's deployment manifests were packaged as an OCI artifact and pushed to the Zot registry using flux push artifact, and then signed with Notation.
2. Flux Deployment with Verification: A Flux OCI repository source and a Kustomization resource were configured to instruct Flux to pull the application manifests from the Zot registry. Critically, this configuration included a verifier stanza, telling Flux to verify the signature of the OCI artifact (the application manifests) using Notation against a secret containing the trusted certificate.
3. Live Verification Failure and Success: The initial deployment attempt failed because the necessary Kubernetes Secret containing the Notation trust configuration was missing. This elegantly demonstrated the verification in action—Flux refused to deploy the application because it could not verify the authenticity of the manifests. Once the secret was created, Flux successfully verified the signed OCI artifact and proceeded to deploy the emoji vote app.

The demo concluded with the vulnerable 2017 emoji vote app running securely, having gone through mirroring, scanning, patching, and signature verification for both its container images and deployment manifests, all orchestrated through a fully open-source, CNCF-centric pipeline. The Kyverno policies were shown to be actively monitoring, highlighting violations for any images not signed by the specified trusted identity, such as the Kubernetes metric server.

Defensive Implications

▶ Watch: Beyond images: Signing SBOMs and attestations for SLSA (17:00)

The practices demonstrated in this talk offer profound defensive implications for organizations grappling with software supply chain security. Implementing these strategies can significantly enhance an organization's security posture against a wide array of threats:

1. Mitigation of External Registry Risks: By mirroring all external dependencies into an internal, controlled OCI registry (like Zot), organizations gain independence from third-party registry rate limits, unexpected changes, or outright disappearance (e.g., GCR to registry.k8s.io migration). This centralizes control and ensures consistent access to critical artifacts.

1. Enhanced Vulnerability Management: Integrating scanning (Trivy) and patching (Copa) directly into the mirroring and signing pipeline establishes a proactive defense against known vulnerabilities. This ensures that even third-party images are remediated before they enter the production environment, significantly reducing the attack surface.

1. Authenticity and Integrity Guarantees: Signing all OCI artifacts (container images, Helm charts, Flux configurations, SBOMs) with tools like Notation provides cryptographic attestation of their origin and integrity. This establishes a verifiable chain of custody, making it exceptionally difficult for attackers to inject malicious code or tamper with artifacts without detection.

1. Preventing Unauthorized Deployments: Deployment-time verification using Kubernetes native policy engines like Kyverno acts as a critical choke point. By enforcing policies that require valid signatures from trusted sources and restrict image origins, organizations can prevent compromised or untrusted artifacts from ever being executed in their clusters. This is a powerful defense against supply chain attacks, including those involving compromised registry credentials.

1. Centralized Control and Compliance: Consolidating all OCI artifacts—not just container images—into a single, internal registry simplifies authentication, authorization, and audit trails. This unified approach makes it easier to enforce security policies consistently across all application components and meet regulatory compliance requirements by demonstrating control over the entire software supply chain.

1. "Golden Path" for Developers: By establishing these practices centrally, organizations can guide development teams onto a "golden path" where secure supply chain practices are the default. This shared efficiency reduces the burden on individual teams and promotes a consistent security baseline across all projects.

1. Early Detection of Anomalies: Even in "warn" mode, tools like Kyverno can flag policy violations for unsigned or untrusted images, providing early warnings about deviations from the desired security posture. This allows security teams to investigate and address potential issues before they escalate.

In essence, these defensive measures transform a reactive, fragmented approach to security into a proactive, integrated, and verifiable framework, significantly hardening the cloud-native attack surface.

Key Takeaways

• Supply chain security for OCI artifacts is non-negotiable: The focus has shifted beyond just container images to include all OCI artifacts (Helm charts, Flux configs, SBOMs), requiring comprehensive security practices across the entire software supply chain.
• CNCF projects offer robust, integrated solutions: A complete, end-to-end secure supply chain can be built using open-source tools from the CNCF ecosystem, including Zot (registry), Flux (GitOps), Notation (signing/verification), Copa (patching), ORAS (registry client), and Kyverno (policy enforcement).
• Mirroring and signing are foundational: Mirroring external dependencies to an internal registry and then cryptographically signing all artifacts (images and manifests) provides attestation and a verifiable chain of custody, mitigating risks from external sources and ensuring integrity.
• Verification at deployment is critical: Implementing admission control policies (e.g., with Kyverno) to verify signatures and restrict image sources ensures that only trusted and untampered artifacts are deployed into Kubernetes clusters.
• Trust policies define security boundaries: Granular trust policies specifying trusted certificates and registry scopes are essential for defining what constitutes an "authentic" artifact, allowing for flexible yet secure enforcement.
• Automation is key to scalability: Automating the entire process—mirroring, scanning, patching, signing, and deploying with verification—through pipelines (e.g., GitHub Actions) is crucial for maintaining a secure and efficient software supply chain at scale.

About the Speaker(s)

Jeremy Rickard is a Principal Software Engineer at Microsoft Azure, where his expertise lies in building and publishing various CNCF projects for use within Azure and Microsoft's broader ecosystem. In addition to his role at Microsoft, Jeremy is a distinguished co-chair of SIG Release in the Kubernetes project, demonstrating his deep involvement and influence within the Kubernetes community. His work and insights are informed by extensive experience with cloud-native technologies and a dedication to advancing software supply chain security practices.