Weaving a VEX Feed Through the Kubernetes Project - Adolfo García Veytia, Stacklok

Weaving a VEX Feed Through the Kubernetes Project

Speakers: Adolfo García Veytia, Stacklok

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=oCbJdcy3zzA

Overview

In this insightful KubeCon EU talk, Adolfo García Veytia, a key figure in both Kubernetes Release Engineering and the Open Source Security Foundation (OpenSSF) OpenVEX project, delved into the complex and critical endeavor of creating a comprehensive Vulnerability Exploitability eXchange (VEX) feed for the Kubernetes project. The talk addresses a fundamental challenge in modern software supply chain security: the pervasive issue of vulnerability scanners generating numerous false positives. These false positives arise because scanners often identify vulnerabilities in dependencies without understanding whether the vulnerability is actually exploitable within the specific context of the software product.

VEX emerges as a pivotal solution to this problem, providing a standardized mechanism for software producers to communicate the actual impact of reported vulnerabilities to their consumers. García Veytia highlighted how VEX documents, by explicitly stating whether a vulnerability affects a product, is under investigation, or has been fixed, can dramatically improve the accuracy of vulnerability assessments. This not only streamlines security operations for users but also helps software producers meet increasingly stringent regulatory requirements, such as the European Union's impending Cyber Resilience Act (CRA), which mandates that software products should not be placed on the market with exploitable vulnerabilities.

The talk detailed the "Big Kubernetes VEX Plan," an ambitious multi-stakeholder initiative to integrate VEX generation into Kubernetes' vast and intricate ecosystem. This plan involves leveraging automated tooling like GoVULN, incorporating expert maintainer assessments, and establishing robust processes for signing and publishing VEX data. García Veytia also introduced Vexflow, a novel tool designed to manage the VEX lifecycle through a chat-ops interface within GitHub issues, demonstrating a practical approach to operationalizing VEX for large, distributed open-source projects.

Background

▶ Watch: Introduction and talk agenda (0:00)

The landscape of modern software development is characterized by a heavy reliance on third-party libraries and components. While this accelerates development, it introduces a significant attack surface, as vulnerabilities in these dependencies can propagate into end products. Traditional vulnerability scanning tools are adept at identifying known Common Vulnerabilities and Exposures (CVEs) within a software bill of materials (SBOM) or container image. However, their primary limitation lies in their inability to discern the actual exploitability or impact of these vulnerabilities in the specific context of the consuming application. This often leads to an overwhelming volume of alerts, many of which are false positives, consuming valuable security team resources and creating alert fatigue.

VEX (Vulnerability Exploitability eXchange) was conceived to bridge this gap. It's defined as a system of chronologically ordered documents designed to communicate the impact of a vulnerability on a specific piece of software. In essence, it answers the critical question: "If there's a CVE reported, does it actually affect my software?" A VEX document contains statements that pair a software product, a vulnerable component, and a vulnerability with a specific VEX status label. There are four primary labels:

• Under Investigation: Acknowledges the vulnerability's presence but indicates ongoing assessment of its impact.
• Affected: Confirms the vulnerability impacts the product.
• Not Affected: States the vulnerability does not impact the product, often with a justification. This is frequently the most valuable status for reducing false positives.
• Fixed: Indicates the vulnerability has been remediated in the specified product version.

VEX is not a single, monolithic specification but rather exists in several "flavors," often developed by community groups facilitated by organizations like CISA (Cybersecurity and Infrastructure Security Agency). These include VEX integrated within CSAF (Common Security Advisory Framework), VEX within the SBOM formats CycloneDX and SPDX, and OpenVEX. The latter, developed under the OpenSSF, is particularly notable for its design as a lightweight, VEX-only format intended to be easily embeddable in other security attestations.

A significant driver for VEX adoption, as highlighted by García Veytia, is the forthcoming EU Cyber Resilience Act (CRA). This regulation imposes strict requirements on software producers, including a mandate that software products must not be placed on the market if they contain exploitable vulnerabilities. VEX offers a direct mechanism to demonstrate compliance by providing clear, verifiable statements about the non-exploitability or non-applicability of reported CVEs. Beyond regulatory compliance, VEX also facilitates better vulnerability management by providing channels for disclosing fixed vulnerabilities, clearly communicating the impact of vulnerabilities, and detailing remediation strategies.

Key Findings

▶ Watch: How VEX helps suppress false positives (3:25)

The talk unveiled a robust strategy for integrating VEX into the Kubernetes project, a massive undertaking given its scale and numerous sub-projects. The core contribution is the "Big Kubernetes VEX Plan," which aims to construct a comprehensive VEX feed by combining information from multiple trusted sources. This plan is not merely theoretical but is being actively implemented through the development of new tools and processes.

The primary key findings and contributions include:

1. The Multi-Source Kubernetes VEX Feed: The plan outlines a sophisticated VEX feed composed of three distinct but integrated sources:

• VEX for Kubernetes Dependencies: This combines automated analysis from GoVULN (the Go vulnerability analyzer) with human-curated maintainer assessments. GoVULN determines if a reported vulnerability is actually reachable and exploitable within the Kubernetes Go codebase, generating native OpenVEX documents. Maintainers, with their deep code knowledge, can issue signed VEX statements for vulnerabilities that GoVULN might miss or misinterpret.
• VEX from Kubernetes Advisories: Leveraging Kubernetes' status as a CVE Numbering Authority (CNA), this feed will complement existing security advisories. It will provide "negative" VEX statements, clarifying that specific Kubernetes artifacts are not affected by certain CVEs, even if they might appear vulnerable to generic scanners. A crucial future step here is the planned adoption of the OSV (Open Source Vulnerability) format as a foundational data source for generating both CVEs and VEX.
• VEX for Container Base Images: Still under discussion, this potential third feed would address vulnerabilities in the often-heavy base images used by Kubernetes, aiming to filter out irrelevant CVEs.

1. Vexflow: A Chat-Ops Tool for VEX Lifecycle Management: To operationalize VEX generation for a project of Kubernetes' size, García Veytia introduced Vexflow. This innovative tool provides a chat-ops interface within GitHub issues, allowing maintainers to interact with vulnerability reports directly. Vexflow automates the process of opening issues for detected vulnerabilities, capturing maintainer assessments (e.g., via slash commands like /not-affected), converting them into OpenVEX statements, and securely publishing them.

1. Secure and Verifiable VEX Distribution: Vexflow integrates with Sigstore for cryptographically signing VEX attestations, ensuring their authenticity and integrity. These signed attestations are then pushed to the GitHub Attestation Store, providing a centralized and verifiable repository for VEX data. This approach leverages modern software supply chain security practices to build trust in the VEX feed.

1. OpenVEX as the Standard Format: The project's commitment to OpenVEX underscores its focus on a lightweight, VEX-only format that is easily embeddable and consumable by various tools and processes, aligning with the broader OpenSSF ecosystem.

These findings represent a significant step forward in making VEX a practical reality for large open-source projects, addressing the core challenges of scalability, trust, and automation in vulnerability management.

Technical Deep Dive

▶ Watch: Understanding VEX document statuses (affected, not affected) (4:08)

Implementing a comprehensive VEX feed for a project as vast and complex as Kubernetes presents a formidable set of technical and organizational challenges. García Veytia meticulously outlined these hurdles, emphasizing that it's a multi-SIG (Special Interest Group) network effort involving SIG Release, SIG Security, and the Security Response Committee (SRC).

The challenges include:

• Trust and Authorization: Determining who has the necessary in-depth code knowledge to issue trustworthy VEX statements and how to securely authorize them to do so.
• Data Sources: Integrating diverse sources of vulnerability information, from automated scanner outputs to expert human assessments.
• Publishing and Lifecycle Management: Deciding where to host VEX documents, how to manage their updates, and ensure their discoverability by downstream tools.

To address these, the "Big Kubernetes VEX Plan" proposes a layered approach built around three primary VEX feeds:

Feed 1: VEX for Kubernetes Dependencies

This feed focuses on vulnerabilities originating from Kubernetes' numerous software dependencies. It combines two critical sources:

1. Automated GoVULN Analysis: For the Go codebase, the GoVULN (Go vulnerability check) tool is instrumental. GoVULN, an analyzer from the Go project, excels at determining the reachability of vulnerabilities. Unlike generic scanners that merely report the presence of a vulnerable dependency, GoVULN uses compiler analysis to ascertain if the vulnerable code path can actually be triggered in the specific context of the Kubernetes application. Crucially, GoVULN has native OpenVEX support, allowing it to directly output VEX documents using the format openvex flag. This provides an automated baseline for VEX generation, particularly effective for Go modules.

1. Maintainer Assessments: While GoVULN is powerful, human expertise remains indispensable. Senior maintainers, possessing deep contextual knowledge of their respective code areas, can issue VEX statements for situations that automated tools might misinterpret or for vulnerabilities in non-Go components. These maintainer-generated assessments are critical for nuanced decisions about exploitability and impact.

Feed 2: VEX from Kubernetes Advisories

Kubernetes functions as a CVE Numbering Authority (CNA), meaning it has the authority to issue its own CVEs. This feed aims to complement these advisories by providing "negative" VEX information. For example, if a CVE is reported that could affect a component like kubelet, but a specific Kubernetes version or configuration makes it non-exploitable, the VEX feed would clearly state that kubelet is Not Affected. This prevents scanners from erroneously flagging artifacts that are, in practice, secure.

A significant future development for this feed is the plan to rewrite Kubernetes advisories using the OSV (Open Source Vulnerability) format. OSV is an incubating vulnerability format within the OpenSSF designed to be granular and machine-readable. By establishing OSV as the canonical source, Kubernetes can then generate various downstream outputs, including:

• The CVE feed (JSON data sent to the CVE database).
• Information for release notes during security releases.
• A dedicated VEX feed through a VEX processor that consumes the OSV data.

Feed 3: VEX for Container Base Images

This feed is currently in the discussion phase. Many Kubernetes container base images are "heavy," pulling in numerous dependencies, which often leads to a high volume of reported CVEs, many of which are not relevant to Kubernetes' actual usage. This feed would aim to apply VEX principles to these base images, filtering out non-exploitable vulnerabilities.

Vexflow: Operationalizing VEX with Chat-Ops

To manage the lifecycle of VEX statements, particularly those from maintainer assessments, the Vexflow tool has been developed. Vexflow operates as a chat-ops interface within GitHub issues, streamlining the assessment and publication process:

1. Vulnerability Detection: Vexflow periodically scans a designated repository for vulnerabilities (either by integrating with OSV scanner or shelling out to others).
2. Issue Creation: Upon detecting a vulnerability, Vexflow opens a GitHub issue, providing triage instructions to maintainers.
3. Maintainer Interaction: Maintainers interact with the issue by commenting with slash commands (e.g., /not-affected "reason"), along with a human-readable justification.
4. Authorization: Vexflow natively supports Kubernetes' OWNERS file mechanism for authorizing maintainers, ensuring that only trusted individuals can issue VEX statements. New maintainers can be authorized via pull requests.
5. VEX Generation and Signing: Vexflow captures the maintainer's comment, converts it into an OpenVEX statement, and then cryptographically signs it using Sigstore.
6. Attestation Publishing: The signed VEX attestation is then pushed to the GitHub Attestation Store, a feature that allows projects to store verifiable attestations linked to their repositories.
7. Issue Closure: Once the VEX statement is published, the GitHub issue is closed.

Assembling the Final VEX Document

The final piece of the puzzle is the vexflow assemble subcommand. This command aggregates all available VEX information:

• It rescans the repository for currently detected vulnerabilities.
• It fetches all relevant VEX statements (signed attestations from the GitHub store, GoVULN outputs) for the specific branch.
• It filters out any VEX statements that no longer apply to active vulnerabilities.
• It then assembles a comprehensive VEX document, combining the vulnerability data with the VEX impact statements. This consolidated VEX document can then be consumed by downstream scanners and tools to accurately suppress false positives.

This intricate architecture demonstrates a forward-thinking approach to managing software supply chain security for a project of Kubernetes' magnitude, ensuring both accuracy and compliance.

Demo / Proof of Concept

▶ Watch: Why VEX matters: CRA and vulnerability management (6:05)

Adolfo García Veytia provided a live demonstration of Vexflow using a deliberately vulnerable Go project called "Insult Connector 2000." This satirical piece of software functions as a proxy connector that, rather than connecting, simply returns insults. Its purpose was to illustrate a common scenario: a vulnerability detected by a scanner that is, in practice, not exploitable within the application's specific context.

The demo proceeded as follows:

1. Initial Scan and Vulnerability Detection: The "Insult Connector 2000" project, despite its simple functionality, was shown to have two vulnerabilities when scanned by an OSV scanner (which pairs with GoVULN). One vulnerability, stemming from a net module dependency, was theoretically exploitable. However, due to the application's design (it never actually connects to the network, only returns insults), this vulnerability was not practically exploitable.

1. Vexflow Issue Creation: García Veytia explained that in a live scenario, Vexflow (running as a cron job or GitHub app) would periodically scan the repository. Upon detecting these vulnerabilities, Vexflow automatically opens GitHub issues for each, providing triage instructions to maintainers.

1. Maintainer Assessment via Chat-Ops: For the net module vulnerability, a maintainer (simulated by García Veytia) interacted with the GitHub issue by adding a comment using a slash command: /not-affected "vulnerable code cannot be controlled by the adversary". A human-readable justification, "This project never connects to the net," was also included. This exemplifies how Vexflow streamlines the process of capturing expert knowledge.

1. VEX Attestation Generation and Signing: After the maintainer's comment, Vexflow processed the input. It converted the comment into an OpenVEX statement, indicating that the specific vulnerability was "Not Affected" with the provided justification. Crucially, Vexflow then signed this OpenVEX attestation using Sigstore, ensuring its cryptographic integrity and authenticity.

1. Publishing to GitHub Attestation Store: The signed OpenVEX attestation was then pushed to the GitHub Attestation Store, a centralized location for verifiable security metadata. García Veytia briefly showed the attestation appearing in the store, although the GitHub UI at the time didn't fully display its content.

1. GoVULN Integration (Second Workflow): A second workflow was demonstrated, showcasing how GoVULN could be run to automatically generate an unsigned VEX document. This output would then be captured by Vexflow, signed, and pushed as another attestation, illustrating the blend of automated and manual VEX generation.

1. Assembling the Final VEX Document: Though time constraints prevented a full live demonstration of the vexflow assemble command, García Veytia explained its function. This command would combine all generated VEX attestations (from maintainer input and GoVULN) with the current vulnerability scan data, producing a comprehensive VEX document. This document, when fed to vulnerability scanners, would enable them to accurately suppress false positives, allowing security teams to focus on truly exploitable issues.

The demo, though concise due to the limited time slot, clearly illustrated the practical workflow of Vexflow and its potential to significantly improve the efficiency and accuracy of vulnerability management in large-scale projects like Kubernetes.

Defensive Implications

▶ Watch: Challenges of implementing VEX for Kubernetes (7:50)

The work presented by Adolfo García Veytia has profound implications for both software producers and consumers in enhancing their defensive postures against software supply chain vulnerabilities.

For software consumers (organizations deploying Kubernetes or other open-source software):

• Reduced Alert Fatigue: The most immediate benefit is the drastic reduction in false positives from vulnerability scanners. By consuming VEX feeds alongside SBOMs, defenders can quickly filter out vulnerabilities that are not actually exploitable or applicable to their specific usage of the software. This allows security teams to prioritize and focus their limited resources on genuine threats.
• Improved Risk Assessment: VEX provides precise context for vulnerabilities. Instead of a generic "CVE-X found," defenders receive a clear "CVE-X is present but Not Affected because of Y," enabling more accurate risk assessments and informed decision-making.
• Enhanced Compliance: For organizations operating under regulations like the EU CRA, consuming VEX feeds will be crucial for demonstrating due diligence and understanding the true security posture of their deployed software.

For software producers (especially open-source projects and commercial vendors):

• Proactive Vulnerability Communication: Producers can proactively communicate the actual impact of vulnerabilities, building trust with their users. This prevents unnecessary panic and costly remediation efforts based on false positives.
• CRA Compliance: VEX offers a clear pathway to demonstrate compliance with regulations like the CRA. By issuing VEX statements, producers can prove that their software, as delivered, does not contain exploitable vulnerabilities, even if underlying components have known CVEs.
• Streamlined Security Workflows: Tools like Vexflow provide a practical, automated framework for managing the VEX lifecycle. Integrating VEX generation into existing developer workflows (e.g., GitHub issues, OWNERS files) minimizes overhead and encourages widespread adoption.
• Leveraging Automation: For Go projects, GoVULN provides a powerful automated mechanism for generating VEX based on reachability analysis. This should be adopted to streamline the process for language-specific ecosystems.
• Secure Distribution: Utilizing Sigstore for signing VEX attestations and distributing them via platforms like the GitHub Attestation Store ensures that VEX data is authentic, tamper-proof, and easily discoverable by consumers. This is vital for building trust in the software supply chain.
• Future-Proofing with OSV: Adopting the OSV format as a foundational vulnerability data source, as Kubernetes plans to do, provides a flexible and machine-readable base for generating not only VEX but also other security advisories, preparing for future security tooling and regulations.
• Community Engagement: Producers should actively engage with initiatives like OpenVEX and the discussions within OpenSSF and SIG Security to contribute to and benefit from evolving best practices in VEX implementation.

In essence, VEX transforms vulnerability management from a reactive, alert-driven process into a proactive, context-aware system. Defenders can move beyond simply knowing what vulnerabilities exist to understanding which ones truly matter, thereby significantly improving their overall security posture.

Key Takeaways

• VEX is Essential for Accurate Vulnerability Management: VEX (Vulnerability Exploitability eXchange) documents provide crucial context about the actual impact of reported vulnerabilities, significantly reducing false positives from security scanners and enabling more efficient security operations.
• EU Cyber Resilience Act (CRA) is a Key Driver: Forthcoming regulations like the CRA mandate that software products must not contain exploitable vulnerabilities. VEX offers a clear mechanism for producers to demonstrate compliance by explicitly stating which vulnerabilities are not exploitable in their context.
• Kubernetes is Building a Multi-Source VEX Feed: The "Big Kubernetes VEX Plan" integrates automated analysis from tools like GoVULN (for Go projects), expert maintainer assessments, and data derived from Kubernetes' own OSV-formatted advisories to create a comprehensive VEX feed.
• Vexflow Operationalizes VEX Through Chat-Ops: Vexflow is a novel tool that streamlines the VEX lifecycle by providing a chat-ops interface within GitHub issues, allowing maintainers to easily assess and publish VEX statements, leveraging Kubernetes' OWNERS file for authorization.
• Secure and Verifiable VEX Distribution is Paramount: Vexflow utilizes Sigstore for cryptographically signing VEX attestations and pushes them to the GitHub Attestation Store, ensuring the authenticity, integrity, and discoverability of VEX data in the software supply chain.
• OpenVEX is the Preferred Lightweight Format: The project emphasizes OpenVEX as the lightweight, VEX-only format for its ease of embedding and consumption, aligning with broader OpenSSF efforts.

About the Speaker(s)

Adolfo García Veytia is a prominent figure bridging the gap between open-source project development and software supply chain security. He serves as a Software Engineer at Carabiner Systems (a new startup) and Stacklok. More importantly for this discussion, he holds two critical leadership roles: he is a Tech Lead on the Kubernetes Release Engineering team (within SIG Release), where he contributes to the release process of one of the world's largest open-source projects. Additionally, he is a Technical Lead for OpenVEX on the Open Source Security Foundation (OpenSSF), an initiative dedicated to improving the security of open-source software. García Veytia brings a unique perspective to the challenges of vulnerability management, combining deep technical expertise in Kubernetes with a commitment to developing open standards for security communication.