Streamlining Vulnerability Management: The Power of VEX Inheritance in Container Ecosystems

Streamlining Vulnerability Management: The Power of VEX Inheritance in Container Ecosystems

Speakers: God, Product Security Tools Team, Nvidia

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=nNlEm_vte2s

Overview

This talk, presented by God from Nvidia's Product Security Tools team at VulnCon, addresses a critical challenge in modern software development: the redundant and inefficient process of managing Vulnerability Exploitability eXchange (VEX) statements, particularly within complex container ecosystems. As organizations increasingly rely on containerized applications, often built upon multiple layers of open-source components, the manual effort required to analyze and attest to the exploitability status of vulnerabilities becomes a significant bottleneck. Nvidia's solution, centered around the concepts of VEX inheritance and Global VEX, aims to streamline this process, enabling organizations to scale their vulnerability management efforts, reduce manual overhead, and accelerate secure software releases.

The core problem tackled is the repetitive analysis of identical open-source software (OSS) packages and their associated vulnerabilities across numerous container images, even those sharing significant common ancestry. By introducing a structured approach to identifying, cataloging, and reusing VEX data from "parent" images, Nvidia demonstrates how to drastically cut down on redundant security analysis. This talk is highly relevant for security practitioners, software architects, and development teams grappling with the complexities of securing vast containerized environments and striving for more efficient compliance workflows.

Background

▶ Watch: The problem: Rewriting VEX for similar container images (1:17)

The proliferation of containerized applications has revolutionized software deployment, offering unparalleled agility and portability. However, this paradigm shift has also introduced significant challenges for vulnerability management. Modern container images are often composites, built from multiple layers, each potentially introducing a myriad of open-source components. Each of these components can carry known vulnerabilities, necessitating thorough analysis to determine their actual impact within a specific product context. This analysis culminates in the creation of Vulnerability Exploitability eXchange (VEX) statements, which provide crucial context regarding whether a detected vulnerability is actually exploitable, mitigated, or not affected in a given product.

Nvidia, having initiated VEX publishing for its NVIDIA AI enterprise container images on the NGC catalog in October 2023, quickly encountered the inherent inefficiencies of this process. Their initial tooling facilitated the syncing of OSS vulnerability reports, the input of VEX status and justification, an approval workflow, and subsequent publication in Cyclone DX format. However, a significant pain point emerged: security teams were repeatedly performing the same VEX analysis for identical open-source components across different builds and releases of what were fundamentally the same or highly similar products.

Initial attempts to address this redundancy focused on leveraging tagging schemas to identify related releases and copy or clone VEX data. This approach, however, proved impractical. Product teams at Nvidia, like many in the industry, often use highly granular and non-sequential tags, such as commit SHAs, making it impossible to reliably infer lineage or commonality based solely on tags. This highlighted the need for a more robust, content-aware mechanism to understand the relationships between container images and their underlying open-source components, thereby enabling intelligent VEX reuse. The existing problem was a lack of a clear, automated method to determine the "lineage" of container images and apply previously established vulnerability analysis to subsequent builds or derived images, leading to significant delays and resource drain in the release cycle.

Key Findings

▶ Watch: New approaches: VEX inheritance and Global VEX introduced (2:39)

The talk introduces two pivotal concepts designed to overcome the challenges of redundant VEX analysis in container ecosystems: VEX Inheritance and Global VEX. These mechanisms form the cornerstone of Nvidia's streamlined vulnerability management strategy.

VEX Inheritance addresses the problem of repeated analysis for common components found across a lineage of container images. The core idea is to identify and catalog "parent images" – defined as the largest reusable subset of open-source components shared among multiple applications or product builds. Once a parent image is thoroughly analyzed for vulnerabilities, and its VEX statements are approved, these statements can be inherited by "final images" that are built upon or incorporate that parent image. This process involves generating a Software Bill of Materials (SBOM) for both the parent and final images, performing a diff to identify newly introduced OSS packages or vulnerabilities, and then only requiring new analysis for these deltas. The goal is to drastically reduce the analysis burden, ensuring that a significant portion of vulnerability assessment is performed once and then reused across all downstream products.

Global VEX, on the other hand, targets a specific, yet common, scenario: the use of custom-patched open-source versions. Many organizations fork open-source projects, apply internal patches to fix vulnerabilities, and then use these custom versions across their product portfolio. Without a centralized mechanism, each instance of this custom-patched component would require individual VEX analysis. Global VEX introduces a policy-driven approach within the scanning tooling to automatically detect these custom OSS versions (e.g., package-version-dash-we-fixed-this). When such a version is identified, a pre-defined VEX status, acknowledging the internal patch, is automatically applied. This mechanism not only reduces noise in vulnerability reports by filtering out already-patched issues but also provides crucial upgrade guidance to development teams, informing them of available internally or vendor-patched versions. Global VEX ensures that vulnerability context for internally managed patches is consistently and automatically applied wherever those patched components are used, across any container image.

Together, VEX inheritance and Global VEX represent a comprehensive strategy to optimize VEX generation and application. VEX inheritance handles the structural reuse of VEX based on image lineage, while Global VEX tackles the specific challenge of custom-patched components, providing a holistic solution for efficient vulnerability management in complex, layered container environments.

Technical Deep Dive

▶ Watch: Understanding Global VEX for custom patched versions (3:59)

Nvidia's approach to VEX inheritance and Global VEX is underpinned by a sophisticated, integrated workflow that spans product cataloging, container scanning, and dedicated VEX tooling. This system is designed to automate as much of the vulnerability management process as possible, shifting security left into the development pipeline.

VEX Inheritance Workflow

The process begins with the identification and registration of approved parent images. A parent image is conceptualized as the "largest subset of open source that is reusable or sharable among multiple applications." This doesn't necessarily mean the base image; it can be any layer in the lineage that encapsulates a significant, common set of OSS components. These parent images are added to Nvidia's product catalog, associated with specific use cases and application groups, ensuring their applicability is well-defined. Crucially, ownership of these parent images is assigned to product teams, as they are best positioned to maintain, test, and update them in response to new vulnerabilities or necessary upgrades. An expiration date mechanism is also built into the catalog to invalidate outdated parent images.

Once registered, a parent image undergoes a rigorous security and compliance process:

1. Scanning: The parent image is scanned by the container scanning infrastructure, generating SBOMs, license reports, and vulnerability reports.
2. License Approval: License reports are reviewed by an Open-Source Review Board (OSRB), with approvals tracked in the product catalog.
3. Vulnerability Analysis & VEX Generation: Security analysts use the VEX tooling to analyze vulnerabilities in the parent image. This involves determining the true exploitability status (e.g., not_affected, fixed, vulnerable) and providing justifications. These VEX statements undergo an approval process by the product owner before they are ready for publication.

When a final image is built, it also goes through the scanning infrastructure. The system automatically detects if the final image was built upon an approved parent image (matching its use case). If a parent image is detected, the system performs an SBOM comparison between the parent and final images. This diff highlights any new OSS packages or vulnerabilities introduced in the final image's layers.

• License Compliance: If no new open-source components are added, the final image can receive automatic OSRB approval, significantly accelerating the release process. If new components are present, only that smaller subset needs to go through the OSRB review.
• Vulnerability VEX Inheritance: For vulnerabilities associated with OSS packages originating from the parent image, the approved VEX statements are inherited. The system is intelligent enough to differentiate: if a vulnerability is tied to an OSS package from the parent image, its VEX is inherited. If it's tied to an OSS package added in the final image, new analysis is required. Special considerations are made for not_affected mitigations that depend on configuration, environment, or dependencies, requiring a quick review to ensure these haven't changed in the final image.

Global VEX Implementation

Global VEX operates in parallel, specifically targeting internally patched or vendor-supplied custom OSS versions. When Nvidia or a third-party vendor provides a patch for an open-source vulnerability, a policy is defined within the scanning tooling. This policy maps the custom version (e.g., package-1.2.3-nvidia-patch) to a specific VEX status (e.g., fixed). During container scanning, if this custom version is detected, the pre-defined VEX statement is automatically applied to any associated vulnerabilities. This significantly reduces noise in vulnerability reports and ensures consistent handling of known patches across the entire product ecosystem. Nvidia aims to automate the ingestion of vendor-supplied VEX details and eventually leverage AI to detect internally developed patches and generate corresponding VEX.

Integrated Tooling and Shift-Left Capabilities

The entire process relies on a tightly integrated ecosystem:

• Product Catalog: A central repository for registering parent and final images, tracking their SBOMs, license approvals, VEX statuses, and ownership. It caches parent image associations for efficient lookup during release.
• Container Scanning Infrastructure: Performs deep scans of container images, generating SBOMs, vulnerability reports, and identifying parent image lineage. It leverages open-source tools like Scoprio and Crane for rapid image metadata analysis (layer digests, size, architecture) without requiring a Docker daemon.
• VEX Tooling: A dedicated application for analysts to input, review, and approve VEX statements. It also integrates with Nvidia's open-source "vulnerability analysis for container scanner agent blueprint" to suggest VEX details, further reducing manual effort.

To "shift compliance left," this entire workflow is integrated into the build pipeline. Before a build, developers can consult the product catalog for approved parent images matching their use case. In-pipeline jobs detect the parent image, verify its approval, and apply the inherited VEX to container scan vulnerability reports. This means developers see only the vulnerabilities that genuinely require triage for their specific additions, not the entire list from the parent image. The system can even block pipelines if non-compliant parent images are used, although Nvidia acknowledges the complexity of implementing such strict gating due to the need for a balanced "net" of available approved parent images.

Limitations and Lessons Learned

Nvidia encountered several limitations during implementation:

• Automated Parent Image Detection: The layer-matching approach (comparing image layer digests) can be unreliable due to varying registry storage mechanisms and compression algorithms. To mitigate this, Nvidia recommends explicitly integrating build configuration to provide parent image details (e.g., parsing Dockerfiles for FROM instructions or ingesting custom build configurations).
• Parent Image Ownership: A significant lesson learned was the initial lack of clarity regarding parent image maintenance. The core product security team expected product teams to own and maintain their parent images, which was not initially understood. Establishing clear ownership for testing, updating, and releasing parent images is crucial for the long-term success of the system.
• Automation Gaps: While much is automated, some manual steps remain, such as manually adding new releases of core AI enterprise parent images to the catalog and manually ingesting vendor VEX details for Global VEX policies. These are key areas for future automation.

Despite these challenges, a pilot launched in November 2023 demonstrated the system's effectiveness for its initial product, confirming the viability and benefits of VEX inheritance.

Demo / Proof of Concept

▶ Watch: Practical steps to implement VEX inheritance (4:40)

While the talk does not feature a live, interactive demonstration in the traditional sense, the entire presentation serves as a detailed exposition of Nvidia's implemented and piloted solution, effectively acting as a proof of concept in operation. The speaker describes a system that has been live for one of their products since a pilot launch in November 2023, demonstrating its real-world applicability and benefits.

The "demo" is conceptualized through the detailed workflow diagrams and explanations of how Nvidia's product catalog, container scanning infrastructure, and VEX tooling interoperate. The key elements demonstrated through this explanation include:

1. Parent Image Registration and VEX Approval: The process of a user/API registering a parent image, its subsequent scanning, generation of license and vulnerability reports, and the manual/semi-automated VEX analysis and approval workflow. This establishes the baseline for inheritance.
2. Final Image Processing and Inheritance: The system's ability to scan a final image, automatically detect its approved parent image based on use case, perform an SBOM comparison, and then inherit approved VEX statements for common components. The resulting reports show only the vulnerabilities requiring new analysis, illustrating the reduction in workload.
3. Shift-Left Integration: How developers in the pipeline receive tailored vulnerability reports, showing only new issues, and how the system can provide early feedback or even block builds based on parent image compliance.
4. Global VEX Application: The conceptual demonstration of how custom-patched OSS versions are automatically identified, and pre-defined VEX statuses are applied, reducing noise in reports.

The speaker's detailed account of the system's architecture, its operational flow, and the lessons learned from its initial deployment underscore that this is not merely a theoretical concept but a functional, deployed solution actively contributing to Nvidia's vulnerability management. The pilot's success validates the core principles of VEX inheritance and Global VEX as effective strategies for streamlining container security.

Defensive Implications

▶ Watch: Importance of ownership and maintenance for parent images (7:00)

The strategies presented by Nvidia for VEX inheritance and Global VEX offer significant defensive implications for organizations struggling with the scale and complexity of vulnerability management in containerized environments. Adopting similar approaches can drastically improve security posture, compliance, and operational efficiency.

1. Reduce Vulnerability Fatigue and Analysis Burden: By inheriting VEX statements from approved parent images, security teams can eliminate the repetitive analysis of known vulnerabilities in common components. This frees up valuable analyst time to focus on truly new or critical vulnerabilities, reducing "vulnerability fatigue" and improving response times.
2. Accelerate Secure Software Releases: Streamlined license and vulnerability approval processes, particularly the automatic approval for final images without new OSS, directly translate to faster release cycles. This "shift left" approach allows developers to identify and address security issues earlier, preventing late-stage bottlenecks.
3. Ensure Consistent Vulnerability Context: Global VEX guarantees that internally developed patches or vendor-supplied fixes are consistently accounted for across all products utilizing those custom components. This prevents false positives from patched vulnerabilities from cluttering reports and ensures that teams are working with accurate exploitability information.
4. Improve Compliance and Auditability: The product catalog, with its detailed tracking of SBOMs, license approvals, VEX statements, and parent image associations, provides a robust, auditable record of security and compliance decisions for every container image. This is invaluable for regulatory requirements and internal governance.
5. Promote Secure Development Practices: By making approved parent images easily discoverable and by providing in-pipeline feedback, the system encourages developers to build upon secure, pre-vetted foundations. This proactive approach inherently raises the baseline security of applications.
6. Centralized Management of Common Components: Establishing clear ownership for parent images ensures that shared components are actively maintained, patched, and updated. This mitigates the risk of "shadow IT" or unmanaged base images lingering in the environment, which can become significant attack surfaces.
7. Enhanced Visibility into Supply Chain: The SBOM comparison functionality provides clear visibility into what open-source components are added at each layer of the container build process. This transparency is crucial for understanding the software supply chain and identifying potential risks.

To implement similar defensive strategies, organizations should:

• Invest in SBOM Generation and Management: A foundational requirement is the ability to generate accurate SBOMs for all container images and manage them in a central catalog.
• Define Parent Image Strategy: Identify common, reusable layers or base images within their ecosystem and establish a process for their approval, VEX analysis, and ongoing maintenance. Crucially, assign clear ownership to product teams.
• Develop VEX Tooling and Workflows: Implement or adapt tooling that supports VEX creation, review, approval, and publication (e.g., in Cyclone DX).
• Integrate Security into CI/CD: Shift vulnerability scanning and VEX application left by integrating these processes directly into the build pipelines, providing immediate feedback to developers.
• Establish Global VEX Policies: For internally patched OSS, create a mechanism to define and automatically apply VEX statements, reducing noise and ensuring consistency.

By adopting these principles, organizations can transform their vulnerability management from a reactive, labor-intensive chore into a proactive, efficient, and scalable security enabler.

Key Takeaways

• VEX Inheritance is Crucial for Scale: Reusing VEX statements from approved "parent images" significantly reduces redundant vulnerability analysis across similar container images, making vulnerability management scalable for complex ecosystems.
• Global VEX Manages Custom Patches: A policy-driven approach automatically applies VEX for internally patched or vendor-supplied custom open-source versions, reducing noise in reports and ensuring consistent handling of known fixes.
• SBOMs are Foundational: Accurate Software Bills of Materials (SBOMs) are essential for identifying common components, performing diffs between parent and final images, and enabling VEX inheritance.
• Integrated Workflow is Key: A cohesive system involving a product catalog, scanning infrastructure, and VEX tooling is necessary to automate VEX generation, approval, and application, shifting compliance left into the development pipeline.
• Ownership and Maintenance are Paramount: Clearly defining ownership for "parent images" by product teams is critical for their ongoing maintenance, testing, and updating, ensuring the continued validity of inherited VEX.
• Automation is an Ongoing Journey: While significant automation is achievable, areas like automatic ingestion of new parent image releases and vendor VEX details remain targets for future enhancements.

About the Speaker(s)

God is a member of the Product Security Tools team at Nvidia, where they have been working for approximately three years. Their expertise lies in developing and implementing solutions to streamline security processes, particularly in the context of container security and vulnerability management. God's work focuses on creating tooling and workflows that enable efficient VEX generation and application, helping Nvidia manage the security of its vast array of container images, especially those related to NVIDIA AI enterprise.