Why Don’t We Have Both? Track Build- and Run-time Information for S... Jeff Mendoza & Ben Hirschberg

Why Don’t We Have Both? Track Build- and Run-time Information for S... Jeff Mendoza & Ben Hirschberg

Speakers: Jeff Mendoza, Software Engineer at Kusari & Guac Maintainer; Ben Hirschberg, Kubescape Maintainer

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=x5qguW0SF_I

Overview

In the rapidly evolving landscape of software supply chain security, understanding the true risk posed by vulnerabilities remains a significant challenge. This talk, "Why Don’t We Have Both? Track Build- and Run-time Information for S...", delivered by Jeff Mendoza and Ben Hirschberg at KubeCon EU, addresses the critical gap between theoretical vulnerabilities identified at build time and actual threats present during runtime. The speakers introduce an innovative approach that combines the comprehensive data aggregation capabilities of Guac (Graph for Understanding Artifact Composition) with the deep runtime insights provided by Kubescape, a Kubernetes security platform leveraging eBPF.

The core problem tackled is the overwhelming volume of "noise" generated by traditional static vulnerability scanners. While these scanners are effective at identifying all packages and potential vulnerabilities within a container image, a significant portion of these identified risks may never actually be loaded into memory or executed during the application's runtime. This leads to severe vulnerability fatigue for security teams, making it difficult to prioritize and remediate the most critical threats. Mendoza and Hirschberg present a solution that filters out these non-actionable vulnerabilities, offering a more precise and actionable view of a system's security posture.

This presentation is highly relevant for anyone involved in Kubernetes security, software supply chain management, or DevSecOps. It provides a practical, open-source framework for gaining deeper visibility into the software running in production, enabling organizations to move beyond mere compliance to genuine risk reduction. By integrating build-time Software Bill of Materials (SBOMs) with dynamic runtime analysis, the talk demonstrates how to achieve a more accurate and efficient approach to vulnerability management, ultimately leading to stronger, more resilient cloud-native applications.

Background

▶ Watch: Speakers introduce Guac and Kubescape projects (0:00)

The journey into understanding software supply chain security often begins with the Software Bill of Materials (SBOM). An SBOM is essentially a nested inventory, a formal record of the components and dependencies used in building a piece of software. While crucial, a standard SBOM typically captures the state of software at a specific point, often build time, or lists all components within a container image. This static view, while foundational, is insufficient for a complete security picture, especially in dynamic environments like Kubernetes.

Guac emerged as an openSSF project to address this limitation. It is a system designed to ingest, aggregate, and query diverse software supply chain metadata. Guac's core is a GraphQL API backed by a graph database, allowing users to connect various pieces of information about software artifacts. Beyond SBOMs, Guac can incorporate SLSA attestations (Supply-chain Levels for Software Artifacts), Scorecard scores (from OpenSSF Scorecard), and VEX documents (Vulnerability Exploitability eXchange). This aggregation allows security teams to understand the complex interconnections between software components, their provenances, and known vulnerabilities, providing a holistic view of the supply chain. Prior to this integration, Guac primarily ingested build-time SBOMs, often generated as a step in the CI/CD pipeline.

On the other side of the equation is Kubescape, a CNCF incubating project that has evolved into a comprehensive Kubernetes security platform. Originating as a CLI tool for scanning Kubernetes clusters and YAML files for misconfigurations, Kubescape expanded significantly. It leverages Open Policy Agent (OPA) for policy enforcement and includes a vulnerability scanner. A key innovation within Kubescape is its eBPF node agent, which enables deep runtime visibility into Kubernetes workloads. This agent performs functions like network policy proposal, seccomp profile management, and anomaly-based runtime detection. While Kubescape's vulnerability scanner identified issues within container images, it faced the same challenge as other static scanners: a high volume of reported vulnerabilities, many of which were not actively exploited or even loaded during runtime.

The confluence of these two projects addresses a pervasive problem in modern software security: vulnerability fatigue. Security teams are constantly inundated with alerts from static scanners, reporting hundreds, if not thousands, of vulnerabilities in container images. The critical issue is that a significant percentage of these vulnerabilities are present in packages that are never actually loaded into memory or executed by the application within the container. This "noise" makes it incredibly difficult for engineers and security professionals to prioritize remediation efforts, leading to wasted time, delayed deployments, and a false sense of security or, conversely, desensitization to real threats. The solution lies in bridging the gap between what's in an image and what's actually used at runtime.

Key Findings

▶ Watch: Guac's architecture: collectors, certifiers, and data flow (2:20)

The central discovery and contribution of this talk revolve around Kubescape's reachability analysis feature and its integration with Guac to provide a more accurate, actionable view of software vulnerabilities. The key findings demonstrate a significant reduction in vulnerability noise and a path towards more effective security prioritization.

Firstly, Kubescape's innovative use of eBPF for runtime monitoring allows it to identify precisely which software packages within a container image are actively loaded into memory or executed by the application. This capability directly addresses the fundamental problem of vulnerability fatigue. By observing file activity and syscalls (like open and execve) during runtime, Kubescape can distinguish between inert components and those critical to the application's operation.

Secondly, this runtime analysis enables the creation of a filtered SBOM. Unlike a traditional SBOM that lists all components in an image, the filtered SBOM includes only those packages that have been "touched" or "reached" during the container's execution. This selective approach drastically reduces the number of reported vulnerabilities. Ben Hirschberg presented compelling statistics, showing a 70% to 90% reduction in identified vulnerabilities in various common applications like Redis, PostgreSQL, and ElasticSearch. For instance, a Redis example saw vulnerabilities drop from 150-170 to around 15, representing nearly a 90% noise reduction. This dramatic decrease means security teams can focus their efforts on a much smaller, more relevant set of risks.

Thirdly, the newly released Kubescape collector for Guac represents a critical integration point. This collector automatically ingests both the full, static SBOMs and the dynamically generated filtered SBOMs from Kubescape into the Guac graph database. This allows security teams to query and analyze these different layers of supply chain data side-by-side. Guac's ability to correlate these diverse SBOMs with other security metadata (like VEX or vulnerability scores) provides unprecedented insights into the exploitability and impact of identified vulnerabilities.

Finally, the talk highlights that different types of SBOMs—source, build, image, and now filtered runtime—offer distinct but complementary perspectives. While source SBOMs reveal all repository dependencies (including dev/test), build SBOMs narrow down to what's compiled, and image SBOMs include OS-level components, the filtered SBOM provides the most precise view of what's actually active in production. This multi-faceted approach, made accessible through Guac, empowers organizations to make informed, risk-based decisions, moving beyond a "fix everything" mentality to a focused "fix what matters" strategy.

Technical Deep Dive

▶ Watch: Kubescape's journey: from CLI to comprehensive security platform (4:40)

The technical architecture underpinning this integrated solution involves the sophisticated interplay of Guac's data aggregation capabilities and Kubescape's runtime analysis, particularly its use of eBPF.

Guac's Architecture:

At its core, Guac functions as a centralized graph database for software supply chain metadata. It exposes a GraphQL API, allowing flexible and powerful queries across interconnected data points. The system comprises several key components:

• Database and Assembler: This is the heart of Guac, storing the graph data (nodes and edges representing artifacts, packages, vulnerabilities, and their relationships) and providing the GraphQL interface.
• Ingestor: This component is responsible for taking raw supply chain documents (like SBOMs, SLSA attestations, VEX documents, Scorecard scores) and transforming them into the graph's nodes and edges, hydrating the database.
• Collectors: These modules are designed to gather data from various sources. Examples include a file collector for ingesting local SBOM files, a GitHub collector that watches repositories for new releases and associated SBOMs, and, crucially, the new Kubescape collector. Collectors either actively pull data or watch for new data events.
• Certifiers: These components enrich the graph with additional security intelligence. For instance, a certifier might query vulnerability databases like OSV.dev to add vulnerability information as nodes and link them to affected packages in the graph. Similarly, Scorecard scores are added by certifiers.

The power of Guac lies in its ability to connect disparate pieces of information. If multiple SBOMs reference the same package, Guac deduplicates and links them, building a rich, interconnected graph that reveals transitive dependencies, build provenance, and associated vulnerabilities across an entire software portfolio.

Kubescape's Reachability Analysis and SBOM Generation:

Kubescape's approach to runtime vulnerability filtering is a multi-step process:

1. Image Detection: The Kubescape operator component, deployed within a Kubernetes cluster, continuously monitors for new container images being deployed. When an unfamiliar image is detected, it triggers the SBOM generation process.
2. Full SBOM Generation: Kubescape's kube-vulnerabilities component (referred to as cubewoolen in the transcript) is responsible for scanning the detected image. It utilizes established open-source tools like Sift and Grype to generate a comprehensive, full SBOM of all packages found within the image. This full SBOM is then stored as a Kubernetes Custom Resource (CR) within the cluster's Kubernetes API server. This makes the SBOM data easily accessible and queryable using standard kubectl commands.
3. eBPF-based Runtime Monitoring: The core of the reachability analysis is Kubescape's eBPF node agent. This agent, deployed on each node, leverages Inspector Gadget (another CNCF sandbox project) to tap into the kernel's eBPF capabilities. It monitors low-level system calls, specifically focusing on file operations (open, execve) within running containers. This allows it to track which files, and by extension, which software packages, are actively accessed or executed by the application during runtime.
4. Filtered SBOM Creation: As the eBPF agent collects runtime data, it correlates this information with the full SBOM stored as a CR. For each package listed in the full SBOM, the agent determines if its associated files were "touched" or "loaded" into memory during the monitored period. Based on this analysis, Kubescape generates a filtered SBOM. This new SBOM CR contains only those packages that were actively used at runtime. The filtered SBOM also provides a clear distinction from the full image SBOM, offering a refined list of potentially vulnerable components. The monitoring typically occurs for a configurable duration (e.g., initial 2-3 minutes, then updates every 10 minutes, stopping after 3-6 hours by default) to capture typical runtime behavior. In cases of eBPF event loss due to extreme CPU overload, Kubescape intelligently stops filtering to prevent generating inaccurate data.

Guac-Kubescape Integration:

The final piece of this technical puzzle is the Kubescape collector for Guac. This collector acts as a Kubernetes client, specifically designed to watch the Kubernetes API server for the Kubescape-generated SBOM CRs (both the full and filtered versions). As these CRs are created or updated by Kubescape, the collector automatically ingests them into the running Guac system. This integration ensures that Guac's graph is continuously updated with the latest build-time and, more importantly, runtime-filtered SBOM information, providing a dynamic and accurate view of the deployed software's security posture.

Demo / Proof of Concept

▶ Watch: Addressing the pain points of vulnerability management (6:50)

Jeff Mendoza walked through a compelling demonstration illustrating the practical benefits of integrating Kubescape's runtime analysis with Guac's aggregation capabilities. The demo showcased how different types of SBOMs provide varying levels of detail and how Guac helps correlate these insights.

The initial setup involved a Kubernetes cluster running with Kubescape, which had already generated full and filtered SBOMs, visible as Custom Resources (CRs) in the Kubernetes API server. Mendoza displayed these CRs using kubectl, highlighting the presence of both the comprehensive image SBOMs and the more concise filtered SBOMs. He also noted that VEX documents could be ingested by Guac but were not the focus of this particular demonstration.

A key part of the demo centered around a simple Go application comprising a server and a job. Both components shared a single go.mod file, indicating common dependencies like gorilla/mux and zerolog. However, the server binary was designed to only use gorilla/mux for HTTP routing, while the job binary exclusively used zerolog for logging. This setup perfectly illustrated how shared dependencies at the source level might not all be utilized by every compiled binary or running container.

Mendoza then navigated to the Guac GraphQL playground, demonstrating how to query the ingested SBOM data. He showed the following:

1. Source SBOM: Generated by OSV Excalibur, this SBOM reflected the entire go.mod file, listing both zerolog and gorilla/mux along with their transitive dependencies. This represented the broadest view of dependencies within the repository.
2. Build SBOM: For the sample server, this SBOM, generated during the build process, correctly showed only gorilla/mux and its transitive dependencies, as zerolog was not compiled into the server's binary. This narrowed the scope to what was actually shipped in the executable.
3. Image SBOM: This was the full SBOM generated by Kubescape for the server's container image. It included the Go packages (e.g., gorilla/mux) and operating system-level packages from the base image (e.g., Wolfie packages like CS certificates bundle and TZ data). This represented everything present in the container.
4. Filtered SBOM (Go Server): For the Go server, the filtered SBOM was not significantly different from the image SBOM in terms of Go packages. This is expected for Go binaries, which are often statically compiled and self-contained, meaning most of their dependencies are loaded directly into the single binary.
5. Filtered SBOM (Debian-based image): To better illustrate the power of filtering, Mendoza showed a filtered SBOM from a Debian-based image. This SBOM was dramatically shorter than its full image counterpart, containing only packages like libSSL, libs, Linux, and TZ data – precisely those components actively used during the container's runtime. This effectively demonstrated the 70-90% noise reduction Kubescape achieves.

Finally, Mendoza briefly showcased the Guac visualizer, a graphical representation of the interconnected graph. While acknowledging its complexity for a quick demo, he highlighted how it visually links images, Go packages, Wolfie packages, and occurrences of artifacts, demonstrating the comprehensive nature of the aggregated data.

The demo effectively conveyed the core message: different SBOMs provide different perspectives, each valuable. The filtered SBOM, however, offers the most precise and actionable view for vulnerability management by focusing solely on runtime-active components. Guac serves as the indispensable platform to bring all these perspectives together for holistic analysis.

Defensive Implications

▶ Watch: Kubescape's reachability analysis using eBPF for vulnerabilities (8:20)

The integration of Kubescape's runtime reachability analysis with Guac's comprehensive supply chain graph has profound defensive implications for organizations managing Kubernetes environments. This approach moves beyond theoretical risk assessment to provide practical, actionable intelligence, significantly enhancing an organization's security posture.

1. Prioritized Vulnerability Remediation: The most immediate and impactful benefit is the drastic reduction in vulnerability fatigue. By focusing on vulnerabilities present in packages that are actually loaded and executed at runtime (as identified by the filtered SBOM), security teams can prioritize remediation efforts on genuine threats. Instead of chasing hundreds of non-actionable alerts, they can concentrate on the critical few, leading to more efficient use of security resources and faster mitigation of real risks. This is critical for meeting compliance requirements and reducing the attack surface effectively.

1. Improved Incident Response: In the event of a security incident or the discovery of a new critical vulnerability (e.g., a zero-day), Guac's aggregated graph, including runtime data, allows for rapid impact analysis. Security teams can quickly query Guac to identify which running workloads are actually affected by a specific CVE, based on their filtered SBOMs, rather than relying on broader, less accurate static scans. This enables targeted patching and containment strategies, minimizing downtime and potential damage.

1. Optimized Container Image Hygiene: The insights gained from filtered SBOMs can be fed back into the development lifecycle to improve container image hygiene. If Kubescape consistently reports that certain packages in a base image are never used at runtime, developers can be empowered to remove these unnecessary components. This reduces the overall size of container images, decreases the attack surface, and minimizes the number of potential vulnerabilities that need to be tracked, even if they are not actively exploited. The mention of Grafana's team improving their images based on Kubescape's findings underscores this practical benefit.

1. Enhanced Policy Enforcement: Guac's ability to aggregate diverse supply chain data can be leveraged with policy engines like Open Policy Agent (OPA) for more intelligent and context-aware security policies in Kubernetes. For example, a policy could dictate that no workload with a critical vulnerability in a reachable package is allowed to deploy, or that specific network policies are automatically applied to pods running vulnerable, reachable components. This moves beyond simple static checks to dynamic, risk-based policy enforcement.

1. Continuous and Dynamic Risk Assessment: The integration provides a mechanism for continuous risk assessment that adapts to changes in both the software and its operational context. As applications evolve or their usage patterns change, Kubescape's eBPF agent will update the filtered SBOMs, and Guac will reflect these changes in real-time. This ensures that the security posture remains accurate and up-to-date, providing a living, breathing view of an organization's software supply chain security.

1. Trust and Transparency: By providing a clear, verifiable record of components and their runtime usage, this approach fosters greater trust and transparency throughout the software supply chain. Stakeholders can have higher confidence in the security claims of applications, backed by concrete evidence of runtime behavior.

In essence, this combined solution empowers defenders to shift from a reactive, overwhelmed stance to a proactive, focused, and data-driven security strategy.

Key Takeaways

• Runtime analysis drastically reduces vulnerability noise: Kubescape's eBPF-driven reachability analysis accurately identifies software packages loaded into memory or executed at runtime, leading to a 70-90% reduction in reported vulnerabilities compared to static image scans.
• Guac provides a holistic supply chain view: Guac acts as a central graph database, aggregating diverse software supply chain metadata including source, build, image, and now runtime-filtered SBOMs, along with SLSA attestations, Scorecard scores, and VEX documents, enabling comprehensive querying and analysis.
• Integration offers actionable security insights: The new Kubescape collector for Guac seamlessly ingests both full and filtered SBOMs into Guac, allowing security teams to correlate build-time information with runtime usage for precise vulnerability prioritization.
• Different SBOMs serve different purposes: Understanding the distinctions between source, build, image, and filtered SBOMs is crucial for effective security. While source SBOMs reveal all repository dependencies, filtered SBOMs highlight the most critical, actively used components in production.
• Empowers focused remediation and image optimization: By identifying truly active components, organizations can prioritize fixing critical vulnerabilities that pose real threats and optimize container images by removing unused, unnecessary packages, thereby reducing the attack surface.
• Open-source collaboration drives innovation: The solution highlights the power of open-source projects like Guac, Kubescape, Inspector Gadget, Sift, and Grype working together to deliver advanced security capabilities for cloud-native environments.

About the Speaker(s)

Jeff Mendoza is a Software Engineer at Kusari, a software supply chain security startup. He is a prominent maintainer of Guac, an openSSF project licensed under Apache 2.0, and is actively involved in several other openSSF projects, as well as the ClearlyDefined project. His expertise lies in building systems that aggregate and analyze software supply chain data to provide actionable security insights.

Ben Hirschberg is a maintainer of Kubescape, a CNCF incubating project that functions as a comprehensive Kubernetes security platform. With a background primarily in software engineering and the security product side of the industry, Ben contributes significantly to the evolution of Kubescape, focusing on its capabilities for configuration scanning, vulnerability management, and advanced runtime detection using eBPF.