Mind the Gap: Bridging Supply Chain Policy With Git-less GitOps... Michael Lieberman & Andrew Martin

Mind the Gap: Bridging Supply Chain Policy With Git-less GitOps... Michael Lieberman & Andrew Martin

Speakers: Michael Lieberman, Co-founder and CTO, Cusari; Andrew Martin, CEO, Control Plane

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=D21yF0E-v2s

Overview

In an era defined by accelerating software delivery and increasingly sophisticated cyber threats, securing the software supply chain has become a paramount concern for organizations worldwide. This talk, "Mind the Gap: Bridging Supply Chain Policy With Git-less GitOps," delivered by Michael Lieberman and Andrew Martin at KubeCon EU, delves into innovative strategies and open-source tooling designed to address these challenges head-on. The presentation highlights the critical need for robust provenance, verifiable software composition, and efficient vulnerability remediation, particularly in light of emerging regulatory frameworks like the European Union's Cyber Resilience Act (CRA).

Lieberman, co-founder and CTO of Cusari, a tech lead in TAG Security, and a governing board member of the OpenSSF, brings deep expertise in open-source security projects like GUAC and Salsa. Martin, CEO at Control Plane, co-chair (emeritus) of TAG Security, and CISO at Open UK, offers a perspective on cloud-native security and the broader implications of open-source policy. Together, they outline a vision for a more secure software ecosystem, emphasizing the shift from reactive vulnerability management to proactive, data-driven supply chain integrity.

The talk is particularly significant because it tackles the inherent tension between rapid feature delivery and robust security. It posits that while "no such thing as secure software" exists in an absolute sense, organizations can significantly optimize their mean time to remediation (MTTR) and enhance their defensive posture by embracing specific tools and practices. By introducing concepts like Git-less GitOps and leveraging the power of graph databases for artifact composition, Lieberman and Martin offer concrete, actionable insights for developers, security professionals, and compliance officers navigating the complex landscape of modern software supply chains.

Background

▶ Watch: Talk Introduction and Speaker Introductions (0:00)

The landscape of software development and deployment is under unprecedented pressure from escalating supply chain attacks. As highlighted in the talk, incidents like the XZ Utils backdoor, the compromise of polyfill.io, and breaches affecting major vendors like Cisco Duo, alongside research indicating that 58% of UK financial services organizations were victims of supply chain attacks in 2024, underscore a stark reality: traditional security approaches are insufficient. Andrew Martin succinctly states, "there is no such thing as secure software," emphasizing that security will always operate one step behind feature delivery in a competitive market.

This persistent vulnerability has spurred legislative action, most notably the EU Cyber Resilience Act (CRA). The CRA fundamentally shifts responsibility for software security from often unpaid open-source maintainers to the commercial entities that profit from and integrate that software. This means organizations selling products in Europe are now accountable for the security of all components, including their open-source dependencies. While the CRA's implementation details are still evolving, with the Linux Foundation, OpenSSF, Eclipse Foundation, and Open UK actively working to define reasonable compliance baselines, its intent is clear: to enforce timely patching and greater accountability. The challenge, however, is immense, as all layers of the software stack—from ingested dependencies and application code to operating system base images, container layers, binary artifacts, underlying platforms like Kubernetes, hardware, and firmware—are perpetually at risk. Even AI-generated code is added to this list of potential vulnerabilities.

To navigate this complex environment, organizations must optimize their mean time to remediation (MTTR). The speakers outline key assumptions for a resilient software supply chain: a defined SDLC (Software Development Life Cycle), reliance on a source repository and a build system, and the adoption of GitOps. GitOps, favored for its inversion of control and removal of secrets from direct production access, relies on an append-only system of record (like a non-force-pushable Git log or a Merkel tree, akin to blockchain's underlying structure) for declarative deployments. Within this framework, verifying the provenance of software dependencies becomes critical. Practices like avoiding curl | bash for direct ingestion are paramount. Instead, organizations should verify software origin using tools like Sigstore and The Update Framework (TUF), ensure secure build processes with Salsa, and track interstitial build steps with in-toto. Building from source and utilizing trusted package repositories (e.g., Nix) are also recommended.

A central theme is the role of Software Bill of Materials (SBOMs). While essential for describing software composition in an ecosystem-agnostic way, SBOMs are not a "panacea." They are mostly useless if merely stored without analysis. Overloading SBOMs with excessive, non-standardized metadata also diminishes their utility. Instead, SBOMs must be actively used for analysis and correlated with other data. Furthermore, securing the build system itself is crucial to prevent "SolarWinds-like" attacks. The talk emphasizes Salsa attestations for build framework adherence, Scorecard and Security Baseline from the OpenSSF for project security posture, and in-toto for cryptographically signing build steps. These practices are being proposed as a valid baseline for CRA self-attestation, highlighting the convergence of technical best practices with regulatory compliance.

Key Findings

▶ Watch: Recap of supply chain attacks and real-world examples (2:26)

The talk presents several key findings and innovative approaches to bridging the gap in supply chain policy and practice:

1. GUAC (Graph for Understanding Artifact Composition) as a Centralized Intelligence Hub: An OpenSSF project, GUAC is introduced as a transformative tool for ingesting, correlating, and querying all software supply chain metadata. It unifies information from SBOMs, VEX documents, Scorecard results, and Salsa attestations with upstream data like license information from ClearlyDefined and vulnerability details from OSV. This creates a powerful graph database that allows organizations to quickly answer critical questions, such as "where does this vulnerability live?" and "does it impact one project or a hundred?", significantly reducing vulnerability assessment toil.

1. Git-less GitOps with OCI Artifacts: A major contribution from Control Plane, this new open-sourced Flux capability fundamentally rethinks how configurations are deployed in GitOps. By using OCI (Open Container Initiative) artifacts in place of traditional Git repositories for deployment, it removes direct network routes from production environments to external Git providers (GitHub, GitLab), reduces the burden of cloning entire repositories, and enables universal security controls (like signing and provenance tracing) for configuration, treating it with the same rigor as application binaries.

1. Hardened CI/CD Pipeline for Comprehensive Data Collection: The speakers advocate for a robust CI/CD pipeline that systematically generates and logs critical security metadata at every stage. This includes generating SBOMs (e.g., in SPDX JSON format using Trivy) at multiple build phases, creating VEX (Vulnerability Exploitability Exchange) documents to contextualize CVEs (e.g., declaring a CVE as not applicable or mitigated), and producing Salsa attestations. All this data is fed into GUAC, creating a queryable, organization-wide understanding of artifacts, essential for rapid response to zero-day vulnerabilities like Log4Shell.

1. Optimized Remediation through Data-Driven Insights: The unified, queryable dataset generated by GUAC empowers security teams and incident responders with a single registry of all dependencies. This allows for immediate identification of affected components and targeted patching, drastically reducing the "toil, rework, and vulnerability assessment" typically associated with wide-ranging supply chain compromises.

1. Leveraging Common Expression Language (CEL) for Advanced Policy Enforcement: The integration of CEL into Kubernetes admission control and Flux is highlighted as a powerful mechanism for low-level, fine-grained policy interrogation and enforcement. This enables advanced security postures, such as multi-tenancy lockdowns and escalation prevention, where policies can dynamically respond to attempted privilege escalations, even from potentially compromised configuration sources.

1. Emphasis on Provenance and Trust: The talk consistently underscores the importance of verifying the provenance of all software. Tools like Sigstore, TUF, Salsa, and in-toto are presented as essential for ensuring that software originates from expected sources and has not been tampered with during its lifecycle.

Technical Deep Dive

▶ Watch: CRA shifts security responsibility to software sellers (4:40)

The core of the talk's technical contribution lies in its proposed architecture for a truly secure and auditable software supply chain, centered around GUAC and the innovative Git-less GitOps approach.

GUAC Architecture and Functionality

GUAC (Graph for Understanding Artifact Composition) is an open-source project under the OpenSSF designed to aggregate and correlate disparate pieces of software supply chain metadata into a unified, queryable graph database. Its primary function is to provide a holistic view of artifact composition, enabling deep insights into dependencies, vulnerabilities, and attestations.

• Data Ingestion: GUAC is built to ingest a wide array of data types, including:
• Software Bill of Materials (SBOMs): Standardized formats like SPDX and CycloneDX are parsed to understand component inventories and their relationships.
• Vulnerability Exploitability Exchange (VEX) documents: These provide crucial context for CVEs, indicating whether a vulnerability is applicable to a specific product or mitigated by other means.
• Salsa Attestations: Proofs that a software artifact was built according to specific secure build framework levels (e.g., Salsa Level 3 or 4).
• Scorecard Results: Security health metrics from the OpenSSF Scorecard project.
• Upstream Data: GUAC correlates ingested data with external sources, such as ClearlyDefined for license information and OSV for comprehensive vulnerability data.
• Graph Representation: All this ingested data is transformed into a rich graph model. This graph connects software packages, their dependencies, build processes, vulnerabilities, and attestations across various projects, versions, and environments. This allows for temporal and spatial analysis of the software estate.
• Querying: GUAC exposes a GraphQL API, allowing users to perform complex queries. For instance, one can query to find all projects affected by a specific CVE, trace the provenance of a given binary, or identify common dependencies across an entire organization. This capability is pivotal for optimizing MTTR during a security incident.

Git-less GitOps with OCI Artifacts

This is a significant innovation, open-sourced by Control Plane as part of Flux. Traditional GitOps relies on Git repositories as the single source of truth for declarative infrastructure and application configuration. While powerful, this approach introduces certain security and operational challenges:

• Network Routes: Production clusters often need to pull from Git repositories, potentially introducing direct network routes to external Git hosts (GitHub, GitLab, internal Git servers). This expands the attack surface.
• Cloning Burden: For large repositories or environments with many clusters, cloning entire Git repos (even sparse ones) can be inefficient and resource-intensive.
• Security Controls: Applying consistent, cryptographically verifiable security controls (like signing and provenance) across diverse Git repositories and their contents can be complex and inconsistent.

Git-less GitOps addresses these issues by replacing Git repositories with OCI artifacts stored in an OCI registry (e.g., Docker Hub, Quay.io, or a private registry) as the source of truth for Flux.

• Mechanism: Configuration manifests (Kubernetes YAML, Helm charts, Kustomize overlays) are bundled into OCI artifacts. These artifacts are then signed using tools like Sigstore, ensuring their integrity and verifiable provenance. Flux is extended to pull these signed OCI artifacts directly from a registry.
• Benefits:
• Reduced Attack Surface: Eliminates the need for production clusters to directly access Git hosts, isolating the cluster from potential Git-based supply chain attacks.
• Universal Security Controls: OCI registries and artifacts have a mature ecosystem of security tools. Signing, scanning, and provenance tracing can be uniformly applied to configuration artifacts, just as they are to container images.
• Efficiency: OCI registries are optimized for artifact distribution, potentially offering performance benefits over Git cloning for large-scale deployments.
• Enhanced Provenance: The signing of OCI artifacts provides an immutable audit trail, linking configuration deployments back to their build systems and original sources.
• Control Plane's Open-Sourced Extensions: This initiative includes several enhancements:
• Cluster Sync for Workload Identity: Improves how workloads identify themselves within a cluster.
• Recursive Bootstrap: Simplifies the deployment of complex, multi-cluster architectures (hub-and-spoke, air-gapped environments) from a single command.
• Common Expression Language (CEL) Integration: This is a powerful addition for policy enforcement.

Common Expression Language (CEL) for Policy Enforcement

CEL is a non-Turing complete expression language designed for evaluating policies and rules. Its integration into Kubernetes admission control and Flux is a game-changer for granular security.

• Kubernetes Admission Control: CEL can be used to write custom admission policies that validate, mutate, or reject API requests based on complex conditions. For example, a CEL policy can prevent the creation of pods with specific vulnerable images or restrict resource requests based on namespace labels.
• Flux Integration: With CEL in Flux, policies can be applied directly to the configuration artifacts being deployed. This allows for proactive enforcement of security rules before resources are provisioned in the cluster.
• Escalation Prevention: A key example provided is preventing privilege escalation. If a compromised configuration (even if signed) attempts to deploy a cluster-admin role in a namespace where it's not allowed, a CEL policy can block it. This provides a crucial secondary layer of defense, even if upstream Git or OCI artifact signing mechanisms are temporarily breached.
• Multi-Tenancy Lockdown: CEL policies are instrumental in enforcing strict multi-tenancy boundaries, ensuring that tenants cannot interfere with each other's resources or escalate privileges beyond their designated scope.

Hardened CI/CD Pipeline

The technical deep dive culminates in a vision for a hardened CI/CD pipeline that systematically collects and utilizes security metadata:

1. SBOM Generation: At various stages (source, build, deploy), Trivy (or similar tools) generates SPDX or CycloneDX SBOMs in JSON format.
2. VEX Document Creation: As vulnerabilities are identified, VEX documents are created to provide exploitability context.
3. Salsa Attestations & in-toto Signing: Build processes generate attestations of their security level (Salsa) and cryptographic proofs of each build step's integrity (in-toto).
4. Vulnerability Scans: Automated scans are performed.
5. Data Ingestion into GUAC: All generated SBOMs, VEX documents, attestations, and scan results are fed into GUAC. This creates the comprehensive, queryable graph that is the foundation for rapid vulnerability response.

This integrated approach ensures that every change, every dependency, and every build step is documented and verifiable, enabling a "single registry of all those dependencies" for responders during a crisis.

Demo / Proof of Concept

▶ Watch: Comprehensive list of what software is vulnerable (7:54)

The live demonstration aimed to illustrate how GUAC, combined with a robust CI/CD process, enables rapid identification and conceptual remediation of vulnerabilities. Despite some Wi-Fi related network connectivity challenges during the live session, the intent and steps were clearly articulated.

The demo began with a Kubernetes cluster running GUAC and a simple application called podinfo. The goal was to artificially introduce a CVE and then demonstrate the process of detecting it and preparing for remediation.

1. Initial Setup and SBOM Generation: The first step involved inspecting the podinfo application running on the cluster. Using Trivy, the de facto open-source security tool, an SPDX SBOM was generated in JSON format for the podinfo binary. This SBOM details all components and their relationships within the application.
2. Uploading SBOM to GUAC: The generated podinfo SBOM was then pushed into GUAC. GUAC ingested this data, parsed it, and integrated it into its giant graph database. This process allows GUAC to connect various software projects over time and space, enabling the discovery of common vulnerabilities and dependencies across an entire software estate. Michael Lieberman explained that with many SBOMs, GUAC can show what got fixed or not fixed in new versions.
3. Identifying a Vulnerable Dependency (Viper): The speakers simulated receiving a security notification about a vulnerability. To demonstrate how GUAC helps, they searched the podinfo SBOM for a specific dependency, Viper (version 1.8.1). This step confirmed that Viper was indeed a dependency of the podinfo application.
4. Artificially Marking as Malicious: In a crucial step to simulate a newly discovered vulnerability, the specific Viper 1.8.1 package was artificially marked as "malicious" within GUAC. This action represents the security team's knowledge of a critical, exploitable vulnerability in that specific component.
5. Querying GUAC for Bad Packages: The next logical step was to query GUAC to identify all instances of the now-marked "bad package" (Viper 1.8.1) across the entire software estate. While the live query faced a temporary network issue, the conceptual flow was clear: GUAC would return all applications or artifacts that depend on the malicious Viper version, providing an immediate, comprehensive impact assessment.
6. Conceptual Remediation Flow: Following the identification, the remediation process was outlined:

• The vulnerability would trigger an automated system (e.g., Dependabot) to create a Pull Request (PR) bumping the dependency to a secure version.
• This PR would be speculatively merged into a temporary branch.
• Automated tests would run to ensure the update does not break the production environment.
• Finally, with human review and a signature, the change would be merged and deployed.

The demo, even with the minor network hiccup, effectively showcased GUAC's potential as a central, queryable source of truth for understanding the impact of vulnerabilities, thereby streamlining the notoriously complex and time-consuming remediation process.

Defensive Implications

▶ Watch: Introducing GUAC: Graph for Understanding Artifact Composition (8:52)

The strategies and tools presented in "Mind the Gap" offer critical defensive implications for organizations aiming to bolster their software supply chain security:

1. Establish GUAC as the Central Source of Truth: Defenders should prioritize integrating GUAC into their security operations. By ingesting all available software supply chain metadata—SBOMs, VEX documents, Salsa attestations, Scorecard results, and vulnerability scan data—organizations can build a unified, queryable graph. This centralized intelligence eliminates the fragmented visibility that often plagues incident response, enabling security teams to rapidly identify the blast radius of new vulnerabilities (e.g., all applications affected by Log4Shell) and initiate targeted remediation efforts. This significantly reduces mean time to remediation (MTTR).

1. Adopt Git-less GitOps for Production Deployments: For critical production environments, transitioning from traditional Git-based GitOps to OCI artifact-based GitOps (using Flux and Control Plane's extensions) is a powerful defensive move. This architectural shift removes direct network dependencies between production clusters and external Git providers, thereby reducing the attack surface. Furthermore, by treating configuration as signed, traceable OCI artifacts, defenders can leverage universal OCI security controls (like image signing and vulnerability scanning) that are already mature for container images, ensuring higher integrity and verifiability for deployed configurations.

1. Harden the CI/CD Pipeline with Comprehensive Attestations: Implement a robust CI/CD pipeline that systematically generates and logs security metadata at every stage. This includes:

• Automated SBOM Generation: Use tools like Trivy to generate SPDX or CycloneDX SBOMs at source, build, and deploy phases.
• VEX Document Creation: Actively create VEX documents to provide crucial context for CVEs, preventing unnecessary remediation efforts for non-exploitable vulnerabilities.
• Salsa and in-toto Attestations: Ensure build processes adhere to secure build frameworks (Salsa) and cryptographically sign each build step (in-toto) to provide an immutable audit trail of provenance and integrity. This makes it brutally difficult for attackers to hide tampering.

1. Implement Granular Policy Enforcement with CEL: Leverage Common Expression Language (CEL) within Kubernetes Admission Control and Flux to enforce fine-grained security policies. This allows defenders to:

• Prevent Privilege Escalation: Block attempts to deploy privileged resources or roles that violate established security boundaries, even if a compromised (but signed) configuration artifact attempts it.
• Enforce Multi-Tenancy: Strictly isolate tenants and prevent unauthorized resource access or modification within shared cluster environments.
• Dynamically Validate Configurations: Ensure that all deployed configurations adhere to security baselines, resource limits, and best practices before they are applied. Tools like OPA and Kyverno can also be used for this purpose.

1. Prioritize Provenance Verification: Move beyond implicit trust. Actively verify the provenance of all ingested software and dependencies using tools like Sigstore and The Update Framework (TUF). This ensures that software comes from expected maintainers and has not been tampered with during transit. Avoid practices like curl | bash in production workflows.

1. Stay Proactive on Regulatory Compliance: Understand the implications of emerging regulations like the EU Cyber Resilience Act (CRA). Actively participate in, or monitor, community efforts (e.g., OpenSSF, Linux Foundation) to define reasonable compliance baselines. The adoption of robust security practices outlined in the talk will naturally aid in meeting these evolving regulatory requirements, framing compliance as a byproduct of good security.

1. Continuous Monitoring and Analysis: SBOMs are not static documents. Integrate them into a continuous monitoring and analysis pipeline. Correlate SBOM data with runtime information (e.g., using tools like CubeCape) to understand actual operational exposure. Establish clear security gates in the SDLC to prevent vulnerable software from reaching production without proper checks and approvals.

By implementing these defensive strategies, organizations can move towards a more resilient, transparent, and auditable software supply chain, better prepared to mitigate the impact of inevitable cyber attacks and comply with evolving regulatory demands.

Key Takeaways

• The EU Cyber Resilience Act (CRA) is a significant regulatory driver, mandating that software creators are responsible for the security of their entire software supply chain, including open-source components.
• GUAC (Graph for Understanding Artifact Composition) is an indispensable tool for consolidating and correlating all software supply chain metadata into a queryable graph database, drastically reducing the mean time to remediation (MTTR) for vulnerabilities.
• Git-less GitOps with OCI artifacts, a new Flux capability, enhances security by eliminating direct Git dependencies from production, reducing attack surface, and enabling universal, signed security controls for configuration.
• A hardened CI/CD pipeline is essential, integrating automated SBOM generation (e.g., with Trivy), VEX documents, Salsa attestations, and in-toto signing to ensure comprehensive provenance and integrity.
• Common Expression Language (CEL), integrated into Kubernetes admission control and Flux, provides powerful, low-level policy enforcement capabilities, enabling multi-tenancy lockdowns and critical privilege escalation prevention.
• Proactive provenance verification (e.g., via Sigstore, TUF) and continuous analysis of software composition are paramount for navigating the evolving threat landscape and meeting regulatory obligations.

About the Speaker(s)

Andrew Martin is the CEO at Control Plane, a company specializing in cloud-native security. He is an esteemed figure in the security community, having served as a co-chair (now an emeritus position) in TAG Security, a technical advisory group focused on cloud-native security within the CNCF. Additionally, he holds the position of CISO at Open UK, an organization dedicated to promoting open source, open hardware, and open data, actively working to prevent regulatory bodies and governments from inadvertently harming the open-source ecosystem, particularly in response to legislation like the CRA.

Michael Lieberman is the Co-founder and CTO of Cusari. He is a prominent contributor to the open-source security landscape, serving as a tech lead in TAG Security. His commitment to open-source security extends to his role as a governing board member of the Open Source Security Foundation (OpenSSF), where he also serves as a TAC member. Michael is a maintainer of several critical open-source projects, including GUAC (Graph for Understanding Artifact Composition) and Salsa (Supply Chain Levels for Software Artifacts), both of which are central to the discussion on supply chain security. He has also co-authored a book on related topics.