Enhancing Software Composition Analysis Resilience Against Contai... Agathe Blaise & Jacopo Bufalino

Enhancing Software Composition Analysis Resilience Against Container Image Obfuscation

Speakers: Agathe Blaise, Research Engineer, Thales; Jacopo Bufalino, PhD Candidate, IMT Atlantique & Researcher, Senam Institute

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=G24upbAXVd8

Overview

In the rapidly evolving landscape of containerized applications, Software Composition Analysis (SCA) tools are indispensable for identifying vulnerabilities within container images. However, a significant challenge emerges from the practice of container image obfuscation, where the contents of an image are intentionally or unintentionally modified in ways that evade detection by these very tools. This talk, presented by Agathe Blaise and Jacopo Bufalino, delves into the resilience of current SCA tools against various obfuscation techniques and proposes a novel approach to enhance their effectiveness.

The presentation builds upon earlier work from KubeCon 2023, which introduced concepts like "malicious compliance" and container obfuscation, demonstrating how simple Dockerfile modifications could dramatically reduce reported vulnerabilities. Blaise and Bufalino expand on this, providing an academic and systematic analysis of how the landscape has evolved, identifying prevalent obfuscation methods, assessing the vulnerability of state-of-the-art SCA tools (both open-source and cloud-based), and analyzing real-world container images for evidence of obfuscation. Their work culminates in the introduction of ORCA, an open-source tool designed to mitigate many of these obfuscation challenges.

The talk underscores a critical disconnect: while developers often employ "best practices" like multi-stage builds and aggressive size reduction to optimize container images, these very practices inadvertently create blind spots for security scanners. This research highlights the urgent need for updated guidelines and more sophisticated SCA methodologies to ensure the transparency and security of containerized environments, preventing critical vulnerabilities from going undetected in production deployments.

Background

▶ Watch: Introduction & the problem of container obfuscation (0:00)

The process of scanning container images for vulnerabilities typically involves two main steps. First, SCA tools index the content of the container image. This involves analyzing the container's file system to identify installed software, including the operating system, OS packages, programming language dependencies, libraries, and binaries. The output of this indexing phase is a Software Bill of Materials (SBOM), which is a comprehensive list of all identified components or packages within the container. Tools like sift from Anchore are commonly used for this purpose, generating SBOMs in formats like JSON, detailing packages with unique identifiers such as Common Platform Enumeration (CPE) and Package URL (PURL), including package name, version, organization, and file paths.

The second step involves vulnerability searching. The generated SBOM is then matched against databases of known vulnerabilities, typically leveraging custom online sources and aggregating them. Tools like gripe, also from Anchore, take the SBOM and report vulnerabilities, often expressed as CVEs (Common Vulnerabilities and Exposures), along with their severity and potential fixes.

SCA tools primarily index specific types of files to identify software:

• Operating System: Explicitly written files like /etc/os-release.
• Package Manager Packages: Database files or metadata specific to package managers (e.g., DPKG for Debian, RPM for Red Hat).
• Programming Language Dependencies: Manifest files like requirements.txt for Python, package.json for Node.js, and installed dependency directories such as dist-info for Python or node_modules for Node.js.
• Binaries: Limited information can be extracted directly from compiled binaries.

By default, these tools analyze the squashed representation of a container image, which combines all layers into a single final file system. They then use regex-based searches to find package-related information. However, this approach has inherent limitations. If critical files are removed (e.g., to minimize image size) or if software is installed from source without a package manager, the necessary metadata for SCA tools might be missing. Furthermore, meaningful content that is not fully indexed, such as full file paths, configuration files, raw binaries (without metadata), Dockerfile history (which might reveal download URLs), or files in intermediate layers that were later removed, presents significant blind spots.

This leads to the concept of container obfuscation: the act of intentionally or unintentionally modifying or generating the content of a container image or designing the Dockerfile in a way that renders installed software partially or totally undetected by SCA tools. While obfuscation can be malicious, this research focuses on unintentional obfuscation, which often arises from common development practices:

• Image Size Minimization: Developers remove installation folders or temporary files to reduce the container's footprint.
• Custom Software Installation: Software is downloaded from source and compiled locally, bypassing standard package managers.
• Multi-stage Builds: Intermediate base images are used, and only select files are copied to the final stage, potentially omitting crucial metadata or dependency files.

These practices, while beneficial for optimizing image size and reducing attack surface, inadvertently make SCA tools vulnerable to obfuscation. The talk aimed to understand these obfuscation types, assess tool resilience, analyze real-world occurrences, and propose countermeasures.

Key Findings

▶ Watch: SCA tools: Matching SBOM against vulnerability databases (4:00)

The research began by establishing a taxonomy of eight distinct obfuscation techniques, derived from academic papers, industry whitepapers, and analysis of open-source software implementation details. These techniques, assumed to be non-malicious in intent, include:

1. OS Obfuscation: Deleting, modifying, or altering files containing operating system information (e.g., /etc/os-release).
2. OS Package Obfuscation: Altering package manager databases or files that contain information on installed OS-level dependencies.
3. Dependency Obfuscation: Modifying programming language dependency files (e.g., requirements.txt for Python, package.json for Node.js).
4. Package Obfuscation: Changing the content of downloaded dependencies (e.g., node_modules) to alter version or author information.
5. URL Obfuscation: Downloading software directly from the internet rather than using official package managers, making it harder to track origins.
6. Alias and Link Obfuscation: Using aliases or symbolic links to hide the real paths of files indexed by SCA tools.
7. Packed Obfuscation: Compressing the content of the image, often through multi-stage builds or tools that merge layers into a single one, losing intermediate layer information.

To assess the resilience of existing SCA tools, the researchers selected six prominent tools: Trivy (CNCF project), Snyk, Docker Scout, Google Artifact Registry, Microsoft Defender for Cloud, and Amazon Inspector. They curated a dataset starting from a base Python 3.10 image and progressively added individual and then combined obfuscation techniques. The results were stark: many tools were vulnerable to individual obfuscation instances, and their effectiveness significantly decreased as more techniques were combined. In numerous cases, obfuscated images reported zero vulnerabilities and zero packages, drastically reducing visibility for security analysts.

A notable observation was that some cloud-based tools (e.g., Google Artifact Registry, Microsoft Defender for Cloud, Amazon Inspector) would refuse to scan images entirely if specific expected files were missing. This creates a critical blind spot, as it becomes difficult to distinguish between an unscannable container and one with genuinely no vulnerabilities in the UI. Furthermore, the number of vulnerabilities and packages reported varied widely across tools, even for the same obfuscated image, highlighting inconsistencies in their detection capabilities. For example, sift reported 9 vulnerabilities for OS obfuscation but 625 for alias obfuscation, indicating different underlying detection mechanisms and sensitivities.

The study then investigated the prevalence of obfuscation "in the wild" across six diverse datasets: DockerHub official images, DockerHub Bitnami images (popular in hobbyist and cloud environments), DockerHub verified open-source software, Quay.io registry, and Amazon ECR registry. A methodology was devised to detect obfuscation by analyzing image layers, reconstructing Dockerfile metadata, tracking file modifications across layers (similar to a Git repository), and identifying instances where software was downloaded from the internet or known dependency files were altered or deleted.

The findings revealed that obfuscation is widespread:

• Over 10% of containers exhibited OS obfuscation, primarily modifying OS information, with some instances of deletion.
• A significant spike in OS package obfuscation was observed, often due to apt update commands, but also linked to installing software from source.
• Roughly 20% of containers involved software downloaded directly from the internet (URL obfuscation).
• Package and dependency removal were also common.

Obfuscation was present across all registries, with only a small, non-significant difference between hobbyist and production-grade containers. Bitnami images, however, showed surprisingly low numbers of OS and OS package obfuscation. This was attributed to their use of a tool called claim that compresses all layers into a single one, effectively obscuring the internal changes and making it difficult for the researchers' methodology to detect specific layer-by-layer obfuscation. This highlights "packed obfuscation" as a particularly challenging technique for analysis.

Technical Deep Dive

▶ Watch: Specific files and methods used by SCA tools (4:50)

The core of the research involved a systematic approach to understanding and detecting container obfuscation. The taxonomy of eight obfuscation techniques provided a structured framework for analysis. For instance, OS obfuscation might involve simply deleting /etc/os-release or modifying its content to misrepresent the operating system. Dependency obfuscation could be achieved by using a COPY --from=builder command in a multi-stage build that only transfers compiled binaries, leaving behind requirements.txt or package.json in an earlier, discarded layer. URL obfuscation is particularly insidious, as software downloaded directly via wget or curl and then compiled avoids package manager records entirely.

To detect obfuscation in the wild, the researchers developed a robust methodology. First, they extracted the container image layers and, leveraging the OCI (Open Container Initiative) standard, reconstructed metadata to infer the original Dockerfile commands. This allowed them to perform a git-like analysis of file system changes across layers. By examining each layer, they could track modifications, additions, and deletions of individual files. For example, if a requirements.txt file was present in one layer but deleted in a subsequent one, it signaled potential dependency obfuscation. Similarly, identifying RUN commands in the reconstructed Dockerfile that involved curl, wget, or git clone pointed to URL obfuscation. This granular, layer-by-layer analysis was crucial because traditional SCA tools often only scan the final, squashed file system.

A practical example of how obfuscation can be leveraged for attacks was illustrated with a popular PostgreSQL Dockerfile. This Dockerfile featured a very large RUN command that first installed dependencies, then downloaded software from the internet, installed PostgreSQL plugins, built the software, and finally downloaded everything. The critical issue here is that the corresponding layer, after this complex build process, would only contain a few binaries. While a Go binary might still be analyzable for dependencies, a C binary compiled from source would not be recognized by any standard SCA tool. This scenario creates a significant blind spot. The speakers cited CVE-2024-10979 as a known vulnerability in PostgreSQL that, if present in such an obfuscated image, could allow an unauthenticated or unprivileged attacker to gain access. Because the SCA tools would not report the vulnerability due to the obfuscated nature of the installation, CI/CD pipelines would fail to flag it, leaving production systems exposed.

To address these challenges, the researchers developed ORCA, an open-source tool designed to enhance SCA resilience. ORCA's mitigation methodology involves a much deeper and more comprehensive analysis than typical SCA tools:

1. Iterative Layer Analysis: Instead of just scanning the squashed image, ORCA performs an iterative analysis of all individual layers and the reconstructed Dockerfile. This allows it to trace the history of file changes and identify artifacts that might have been present in intermediate layers.
2. Expanded Package Metadata Analysis: ORCA doesn't just look for manifest files (requirements.txt). It delves into the actual downloaded packages and their contents, attempting to extract version and author information directly, even if the primary metadata files are altered or missing.
3. Binary and Configuration File Analysis: Recognizing the limitations of metadata-only scanning, ORCA analyzes compiled binaries (e.g., Go binaries can often be inspected for embedded dependency information), configuration files, and libraries.
4. Comprehensive File Path Coverage: ORCA aims to increase the coverage of the container file system by analyzing file paths more thoroughly, searching for patterns that indicate installed software even without explicit package manager records.

ORCA's results demonstrate significantly improved resilience compared to state-of-the-art tools. While it can mitigate many instances of obfuscation, the researchers acknowledged that data loss inherent in certain techniques, particularly aggressive multi-stage builds and highly compressed images, still poses a challenge. When layers are heavily optimized or entirely collapsed, the historical context needed for full reconstruction can be permanently lost.

Demo / Proof of Concept

▶ Watch: Defining container obfuscation and its common causes (7:00)

While the talk did not feature a live, interactive demonstration of the ORCA tool or a real-time exploit, it effectively illustrated the impact of container obfuscation through concrete examples and the presentation of ORCA's capabilities. The primary "proof of concept" was the detailed explanation of the PostgreSQL vulnerability scenario (CVE-2024-10979) within an obfuscated image. This served to highlight how common Dockerfile practices, leading to custom compilation and binary-only layers, could effectively hide critical vulnerabilities from standard SCA tools, leaving systems exposed without any security alerts.

The speakers presented the results of ORCA's performance against their curated dataset of obfuscated images, showing its superior ability to detect packages and vulnerabilities compared to commercial and open-source alternatives. They explained ORCA's deep analysis methodology, including its iterative layer analysis and broader file system coverage, which are the technical underpinnings of its enhanced detection capabilities. Although no live demo was conducted, the detailed architectural explanation and comparative results effectively conveyed the tool's functionality and its potential to significantly improve container security posture.

Defensive Implications

▶ Watch: Project objectives: Understanding, studying, analyzing, proposing counter-mea... (8:15)

The findings from this research present several critical implications for both developers building container images and security teams responsible for scanning them. The pervasive nature of unintentional container obfuscation means that current SCA strategies are often insufficient, creating hidden vulnerabilities that could be exploited in production.

For SCA tool vendors and security researchers, the primary implication is the urgent need to evolve beyond simplistic, squashed-image analysis and regex-based searches. Tools must adopt more sophisticated techniques, such as:

• Layer-by-layer analysis: Inspecting each layer independently to track file modifications and deletions, as demonstrated by sift all and ORCA.
• Dockerfile reconstruction and analysis: Inferring build steps and dependency sources from Dockerfile metadata.
• Binary analysis: Developing improved capabilities to extract information from compiled binaries, especially for languages like C/C++ where package managers are often bypassed.
• Broader file system coverage: Extending searches beyond known metadata files to include configuration files, libraries, and other indicators of installed software.
• Enhanced SBOM generation: Creating more comprehensive SBOMs that capture the history of changes and potential sources of obfuscation.

For developers and DevOps teams, the research highlights a tension between optimizing image size and ensuring scan transparency. While practices like multi-stage builds are valuable for reducing image footprint and attack surface, they must be implemented with an awareness of their potential to obfuscate. New "best practices" should be considered:

• Prioritize transparency: While reducing image size is important, it should not come at the cost of making the image unscannable.
• Metadata retention: Explore mechanisms to embed or standardize additional metadata during the build process that explicitly lists installed components, even if the original installation files are removed. This could involve new OCI annotations or SBOM standards.
• Avoid aggressive file deletion: Be mindful of deleting critical metadata files or package manager databases.
• Use robust SCA tools: Integrate advanced tools like ORCA into CI/CD pipelines to catch obfuscated components.
• Audit Dockerfiles: Regularly review Dockerfiles for practices that could lead to obfuscation, such as direct software downloads (RUN curl ...) without subsequent package manager registration.
• Pin versions: Ensure all dependencies are explicitly version-pinned in manifest files to prevent unexpected vulnerability changes.

The challenge of "packed obfuscation" (e.g., Bitnami's use of claim to squash layers) suggests a need for industry-wide discussions on how to manage highly optimized images without losing critical security context. The development of ORCA demonstrates a viable path towards mitigating many of these issues, offering a more resilient approach to container image analysis. Ultimately, a community-wide effort is required to update container building guidelines, balancing performance and security to ensure that critical vulnerabilities are always detectable.

Key Takeaways

• Container image obfuscation is a significant and ongoing problem in 2025, impacting the accuracy of Software Composition Analysis (SCA) tools.
• Obscured images are widespread across various use cases, from hobbyist projects to production-grade deployments, posing a risk to diverse containerized environments.
• State-of-the-art SCA tools are vulnerable to these obfuscation techniques, with many failing to detect installed software or vulnerabilities when images are intentionally or unintentionally modified.
• Tools like ORCA offer enhanced resilience, leveraging iterative layer analysis and comprehensive file system scanning to discover and mitigate many obfuscation cases, though some data loss from multi-stage builds and compression remains a challenge.
• Current "best practices" for container image optimization (e.g., multi-stage builds, aggressive size reduction) inadvertently contribute to obfuscation, creating a critical imbalance between image efficiency and scan transparency.
• There is an urgent need for updated guidelines and community effort to develop new container building practices that balance image size reduction and vulnerability mitigation with the imperative for easy and transparent security scanning.

About the Speaker(s)

Agathe Blaise is a Research Engineer working at Thales in France. Her research focuses specifically on the security of virtualized network environments. Her work contributes to projects like SE for Y for Sec, a European initiative aimed at enhancing cybersecurity.

Jacopo Bufalino is a PhD candidate at IMT Atlantique University and also works as a researcher at the Senam Institute in Paris. His research primarily revolves around container and network security, with a particular interest in the challenges posed by container image obfuscation.