BOF - Discussion Regarding False Positive Results from Vulnerability Scanners and the Use of VEX

BOF - Discussion Regarding False Positive Results from Vulnerability Scanners and the Use of VEX

Speakers: Lisa, Microsoft; Pete, Red Hat

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=TFT_8oHAYSw

Overview

This Birds of a Feather (BoF) session, facilitated by Lisa from Microsoft and Pete from Red Hat, delved into the pervasive problem of false positives generated by vulnerability scanners and explored the critical role of Vulnerability Exploitability eXchange (VEX) as an industry-wide solution. The discussion brought together producers, scanning vendors, and enterprise consumers to collectively address the fragmentation and lack of trust plaguing current vulnerability management practices. The core premise is that VEX, if standardized and widely adopted, can significantly reduce the noise in vulnerability reports, allowing organizations to focus on actual risks and improve automation.

The talk underscored that the current ecosystem, where scanner vendors primarily rely on generic data sources like NVD and often lack specific vendor context, leads to an overwhelming volume of alerts that do not reflect the true exploitability of vulnerabilities within a given product. This creates a substantial burden on software producers to repeatedly explain why their products are not affected by reported CVEs, and on consumers to sift through irrelevant findings. The speakers and participants passionately advocated for a unified approach to VEX, emphasizing the necessity of common standards, robust tooling, and a centralized distribution mechanism to transform reactive vulnerability management into a more efficient and trustworthy process.

The conversation highlighted that while the concept of VEX is gaining traction, its effective implementation hinges on overcoming significant challenges, including a proliferation of emerging standards, internal organizational hurdles in data aggregation, and the need for collective industry investment in shared infrastructure. Ultimately, the session served as a call to action for collaboration, aiming to establish VEX as a foundational element for a more secure and rational software supply chain.

Background

▶ Watch: Introduction: Call for VEX industry synchronization and adoption (0:00)

The landscape of vulnerability management has long been characterized by a fundamental disconnect between the identification of a software vulnerability and the assessment of its actual impact on a specific product or deployment. For a considerable period, a vulnerability has been conflated directly with risk by end-users, an oversimplification that often loses critical context. This problem is exacerbated by the current data flow in the industry, which typically involves:

1. Producers: Software vendors who create and maintain products.
2. Aggregators: Entities like the National Vulnerability Database (NVD) and CVE, which serve as central repositories for vulnerability information.
3. Scanning Vendors: Companies that develop tools to scan software for vulnerabilities, often relying heavily on aggregated data.
4. Consumers: Enterprise-level organizations that use software and rely on scanner outputs for risk assessment and patching decisions.

A major pain point discussed was the reliance of most security scanner vendors on NVD as their primary data source, with estimates suggesting 90% of their data originates from NVD. Other sources include OSVDB and various commercial feeds. While these sources are vital, they often provide generic vulnerability information (e.g., affecting a specific package version) without the nuanced context of how that package is integrated or used within a larger product.

This lack of context frequently leads to false positives. A scanner might detect an older version of OpenSSL within a product and flag a known CVE, even if the product's specific configuration, compilation flags, or internal mitigations render it immune to that vulnerability. As one participant noted, this results in "laundry lists" of issues from customer scanner reports, forcing vendors to repeatedly explain, "We're not vulnerable, we're not vulnerable, we're not vulnerable." This "he said, she said" dynamic wastes significant resources for both producers and consumers, eroding trust in vulnerability reports.

Furthermore, the Common Vulnerability Scoring System (CVSS), while widely adopted, faces criticism because its base score is often "overrun with environmental inputs." This means that while CVSS aims for a standardized severity, its practical application can be inconsistent and difficult for customers to interpret, leading to "a lot of push back from vendors" and a general sense of distrust in the scores provided.

The discussion also touched upon Software Bill of Materials (SBoMs) as a related foundational effort. While SBoMs aim to provide a comprehensive list of components within a software product, their current adoption primarily focuses on compliance rather than active vulnerability management. As highlighted by a Cisco representative, ingesting SBoMs for vulnerability management is still a rarity, with most organizations using them simply as a "check to block compliance." This underscores the broader challenge of translating raw component data into actionable security intelligence.

The overall sentiment was that the industry operates in a fragmented state, with a lack of common standards for communicating vulnerability exploitability. This fragmentation, characterized by "too many standards to choose from" (e.g., OpenVEX, CSAF VEX), inhibits the development of robust, interoperable tooling and prevents the widespread adoption of solutions like VEX, perpetuating the existing inefficiencies and distrust.

Key Findings

▶ Watch: Real-world problem: Customer pushback on CVSS base scores (2:00)

The VulnCon BoF session on VEX and false positives yielded several critical findings and a strong consensus on the future direction for vulnerability management:

• VEX is Indispensable for Accurate Vulnerability Assessment: The central finding is that VEX (Vulnerability Exploitability eXchange) is not merely a desirable feature but a crucial mechanism for addressing the epidemic of false positives. By enabling software producers to publish authoritative attestations on whether a known vulnerability affects their specific product, VEX provides the essential context missing from generic vulnerability databases. This allows consumers to quickly determine their actual risk posture, moving beyond the simplistic "package X version Y is vulnerable" approach.

• Industry Standardization is Paramount for VEX Adoption: A recurring theme was the urgent need for a common, widely accepted standard for VEX. With multiple emerging formats like CSAF (Common Security Advisory Framework) VEX and OpenVEX, the industry faces "too many standards to choose from." This fragmentation hinders the development of universal tools (e.g., "robust VEX readers") and prevents seamless data exchange between producers, scanning vendors, and consumers. Without a unified approach, VEX risks remaining a niche solution rather than an industry-transforming one.

• Strong Desire for a Centralized, Authoritative Source: A significant finding was the collective aspiration for a single, centralized platform for vulnerability information, including VEX attestations. Participants envisioned an enhanced CVE.org that would serve as the "single source of truth," where every vendor could push their vulnerability data and VEX statements. This would eliminate the current burden on scanner vendors to pull data from "thousands of different vendors" and fragmented sources, streamlining the entire ecosystem. Such a platform, while posing immense funding and infrastructure challenges, was seen as the only long-term solution to avoid "perpetuating the problem" for another decade.

• Current Scanning Practices Are Insufficient and Detrimental: The discussion highlighted that scanner vendors, by primarily relying on NVD and general package versions, fail to provide accurate vulnerability assessments. This leads to a high volume of false positives and, conversely, can miss critical vulnerabilities due to a lack of specific vendor analysis. This practice forces customers into a reactive, manual validation process and places an undue burden on producers to respond to erroneous reports, demonstrating that the current model is broken and unsustainable.

• Trust and Automation are Core Challenges: Two intertwined challenges were consistently raised: trust and automation. Consumers need to trust the VEX data provided by producers, which necessitates transparency, clear justifications for "not affected" statements, and mechanisms for feedback. Simultaneously, the volume of vulnerability data demands automation for ingestion, processing, and distribution. Manual processes are simply not scalable, especially when dealing with "thousands of CVEs" daily. VEX's success hinges on building both trust and automated workflows across the supply chain.

• Internal Hurdles to VEX Production are Significant: Even for producers committed to VEX, the internal process of generating these attestations is complex. It involves "hurting the cats" – coordinating across numerous product groups and engineering teams to gather consistent data, conducting product-specific vulnerability analysis (e.g., for OpenSSL in their specific product), and educating internal stakeholders on the "why" behind VEX. This underscores that VEX implementation is not just a technical challenge but also an organizational and cultural one.

Technical Deep Dive

▶ Watch: Identifying key VEX ecosystem players: producers, scanners, consumers (4:30)

The technical discussion revolved around the practical implementation of Vulnerability Exploitability eXchange (VEX) and the challenges of standardization, data flow, and tooling. At its core, VEX is a mechanism for a software producer to communicate the applicability of a known vulnerability to their products. This attestation clarifies whether a product is affected, not affected, fixed, or under investigation regarding a specific CVE.

A critical aspect of VEX, often referred to as its "special sauce," lies in the justifications provided for "not affected" statements. As one speaker highlighted, simply stating "not affected" is insufficient; VEX requires specifying why a component is not affected. Common justifications include "inline mitigations" (the vulnerability is mitigated by other code), "component not present" (the vulnerable part of the component is not included), or "configuration prevents exploitation." This detailed context is what differentiates VEX from generic vulnerability data.

The industry is currently grappling with "too many standards" for VEX. Two prominent formats discussed were:

• CSAF (Common Security Advisory Framework): Both Microsoft and Red Hat are actively leveraging CSAF to encapsulate their VEX statements. Microsoft, for instance, has been publishing CSAF files since November and is working towards embedding VEX within them. Red Hat is also using CSAF for its VEX efforts. CSAF provides a structured, machine-readable format for security advisories, making it a natural fit for VEX attestations.
• OpenVEX: Another emerging standard, indicating the nascent and evolving nature of VEX standardization.

The CVE v5 schema was mentioned as already supporting elements akin to VEX, such as identifying components, vulnerability ranges, and basic vulnerable/not vulnerable states. However, it explicitly lacks the rich justification fields that are central to comprehensive VEX. One participant suggested that the CVE system, particularly through ADPs (Authorized Data Publishers), could evolve to incorporate VEX by allowing CNAs (CVE Numbering Authorities) and ADPs to publish specific attestations for their downstream products. For example, OpenSSL as a CNA could release a CVE, and then various ADPs (Fedora, Red Hat, Debian, Ubuntu, Azure Linux) could each publish their own VEX statements for how that OpenSSL CVE affects their specific packages, including justifications.

Microsoft shared its strategy for VEX implementation, adopting a "crawl, walk, run" approach:

1. Start Small: They began with their Azure Linux DRO (Distro), which already formally deals with third-party CVEs. While currently using OVAL files and the Security Update Guide API, the goal is to formalize this process with VEX statements.
2. Publishing Mechanism: They aim to have a VEX directory parallel to their CSAF directory, potentially containing a VEX file for each of the 250,000 CVEs relevant to Azure Linux. Initially, these attestations have focused on "fixed" vulnerabilities, but the next step is to include "not affected" statements with justifications to eliminate "red flags from a scanning vendor."
3. Integration into Build Systems: For broader adoption across Microsoft, they plan to link an attestation tool into their common build system. This would act as a "barrier to the build," requiring engineering teams to attest to a vulnerability's status (not affected with justification, or affected with a fix) before a build can proceed. This ensures VEX generation is integrated into the software development lifecycle.

Red Hat, an early adopter of VEX, also shared their experience, particularly in publishing unaffected CVEs across their entire product corpus. This effort, while significant and internally challenging, provides comprehensive context to customers.

The discussion also highlighted the formidable challenges in producing VEX data:

• Internal Data Aggregation ("Hurting the Cats"): Gathering consistent vulnerability assessment data from diverse product groups within a large organization is a major hurdle. It requires extensive internal education on "what is VEX" and "what structured data do we need."
• Tooling for Production: Developing internal tools to collect, analyze, and format this data into VEX documents is a substantial undertaking.
• Robust Readers: The current lack of "robust VEX readers" that can parse various VEX formats and allow consumers and scanner vendors to "appropriately search it" remains a significant barrier to adoption. Pete from Red Hat noted that while his boss has written one, a community-driven, open-source effort is preferred.
• Funding and Infrastructure for Centralization: The "pie in the sky idea" of a single, centralized database (e.g., an enhanced CVE.org) that contains all vendor-attested vulnerability and VEX data for all software ever would be an immense undertaking. Cisco's SBoM management platform alone is 200GB, and a global VEX database could easily reach "two terabytes or more," requiring substantial collective funding and architectural innovation.

Ultimately, the technical deep dive reinforced that VEX is a powerful concept with clear benefits, but its successful, widespread implementation demands not only technical solutions for formatting and processing but also significant organizational alignment, collective industry investment, and a commitment to standardization.

Demo / Proof of Concept

▶ Watch: Cisco's experience: VEX production is feasible, ingestion is complex (6:50)

As a Birds of a Feather (BoF) session, this talk was designed as an open discussion rather than a formal presentation of new research or a demonstration of specific tools. Therefore, the session did not feature a live demonstration or proof of concept of VEX in action. The speakers and participants engaged in a dialogue, sharing their experiences, challenges, and aspirations regarding VEX implementation and adoption within their respective organizations. While Microsoft and Red Hat described their ongoing internal efforts to produce and consume VEX, these were discussed conceptually rather than through a live technical showcase.

Defensive Implications

▶ Watch: Industry challenge: Waiting for structured VEX data to act (8:40)

The widespread adoption of Vulnerability Exploitability eXchange (VEX) holds profound implications for security defenders, promising to transform how organizations manage and respond to software vulnerabilities.

1. Reduced Noise and Focused Prioritization: The most immediate and significant benefit of VEX for defenders is the drastic reduction in false positives from vulnerability scanners. By receiving authoritative attestations from software producers that a specific CVE does not affect their particular product (along with clear justifications), defenders can eliminate countless hours spent investigating and validating irrelevant alerts. This allows security teams to focus their limited resources on truly exploitable vulnerabilities that pose actual risk, leading to more efficient remediation efforts and improved overall security posture.

1. Enhanced Trust and Clarity in Vulnerability Data: VEX provides a standardized, machine-readable mechanism for producers to convey precise context about vulnerabilities. This directly addresses the "trust issue" highlighted in the discussion, where generic CVSS scores and scanner outputs are often viewed with skepticism. When a vendor explicitly states "not affected" or "fixed" via VEX, backed by specific justifications, defenders gain greater confidence in the accuracy of the information, enabling more informed decision-making.

1. Improved Automation of Vulnerability Management: The machine-readable nature of VEX documents (e.g., CSAF VEX) is crucial for automating vulnerability management workflows. Instead of manually reviewing lengthy scanner reports and vendor advisories, security tools can programmatically ingest VEX data, correlate it with their asset inventory, and automatically update the status of vulnerabilities. This accelerates the identification of critical issues, streamlines the patching process, and allows for quicker validation of mitigation effectiveness. As one participant stressed, "we cannot be reading something that it's nice for a person, but you cannot put it in a system that will automatically create everything."

1. Informed Risk Assessment: With accurate VEX data, defenders can perform more precise risk assessments. Instead of relying on a generalized CVSS base score, they can factor in the specific exploitability context provided by the vendor. This moves beyond simply identifying vulnerabilities to understanding their true impact within the organization's unique environment.

1. Advocacy and Demand for VEX: Defenders, especially enterprise consumers, should actively encourage and demand that their software vendors provide VEX attestations for their products. This collective demand will drive broader adoption and push the industry towards standardization. Engaging with vendors and participating in industry initiatives (like the one proposed by Pete from Red Hat) can accelerate this shift.

1. Shift Towards Proactive Asset Management: While VEX primarily addresses vulnerability assessment, the underlying discussion also touched upon the need for robust Software Bill of Materials (SBoM) and asset management systems. In the long term, a comprehensive, accurate catalog of all software components (via SBoMs) combined with VEX data could fundamentally shift vulnerability management from reactive scanning to a more proactive, continuous assessment model. Instead of scanning to "find what's in my software," defenders could query their asset management system with VEX data to instantly know their exposure.

In essence, VEX empowers defenders to move beyond the current state of "interesting warfare" between producers, scanners, and consumers, fostering a more collaborative and efficient ecosystem where accurate, actionable vulnerability intelligence is readily available and trusted.

Key Takeaways

• VEX is essential for addressing false positives: Vulnerability Exploitability eXchange (VEX) provides critical context from software producers, allowing defenders to distinguish between reported vulnerabilities and those that actually affect their specific product, significantly reducing the noise from scanner outputs.
• Industry-wide standardization is crucial for adoption: The proliferation of different VEX formats (e.g., CSAF VEX, OpenVEX) hinders interoperability and the development of universal tools. A collective effort towards a common standard is necessary for widespread and effective VEX implementation.
• A centralized platform for VEX data is a highly desired long-term goal: There is a strong industry desire for a single, authoritative source (potentially an enhanced CVE.org) where all vendors can publish their VEX attestations, streamlining data consumption for scanner vendors and end-users.
• Producing VEX requires significant internal effort and education: Generating accurate VEX data involves complex internal coordination across product teams, detailed product-specific analysis, and educating stakeholders on the purpose and requirements of VEX.
• Defenders benefit from VEX through reduced noise, improved trust, and automation: VEX enables security teams to prioritize actual risks, gain greater confidence in vulnerability information, and automate vulnerability management workflows, leading to more efficient and effective security operations.
• Collaboration across the software supply chain is vital: The success of VEX hinges on collective effort and investment from software producers, scanning vendors, enterprise consumers, and aggregators to establish common standards, develop robust tooling, and build a trusted, automated data exchange ecosystem.

About the Speaker(s)

The BoF session was co-facilitated by two prominent figures actively involved in the VEX space:

• Lisa, Microsoft: Lisa is a key contributor to Microsoft's VEX initiatives. She is a strong advocate for industry synchronization on VEX standards to ensure consistency and interoperability. Her work at Microsoft includes developing a publishing tool for CSAF files and spearheading the integration of VEX into Microsoft's processes, starting with the Azure Linux DRO. She is actively working to formalize VEX attestations for third-party CVEs and integrate them into Microsoft's build systems, aiming to make VEX a fundamental part of their engineering lifecycle.

• Pete, Red Hat: Pete (identified by his email pal@redhat.com) is deeply involved in Red Hat's VEX efforts, which include publishing attestations for unaffected CVEs. He played a central role in facilitating the discussion, guiding participants through the complexities of VEX adoption, and highlighting the challenges and opportunities for industry collaboration. Pete's contributions underscore Red Hat's commitment to advancing VEX and fostering a community-driven approach to solving the broader problems in vulnerability management. He extended an invitation for interested parties to join a mailing list to continue the collaborative work on VEX standardization and implementation.