Insane in the Supply Chain: Threat modeling for...

Insane in the Supply Chain: Threat modeling for...

Speakers: Eoin Wickens, Marta Janus

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=rUf5g7sDzq8

Overview

This talk, "Insane in the Supply Chain: Threat modeling for attacks on AI systems," delivered by Eoin Wickens and Marta Janus, researchers at Hidden Layer's Synaptic Adversarial Intelligence (SAI) team, delves into the critical and rapidly evolving landscape of security threats targeting the artificial intelligence supply chain. Building upon their previous work introducing attacks on AI and machine learning, this presentation focuses on the specific vulnerabilities and attack vectors that emerge when AI systems are integrated into broader software development and deployment pipelines. The speakers highlight the urgent need for robust threat modeling in an era where AI, particularly Large Language Models (LLMs), is being rapidly adopted across virtually every product and service imaginable.

The core of the discussion revolves around three primary attack surfaces within the AI supply chain: data poisoning, model hijacking and backdooring, and the exploitation of AI/ML tooling and services. The talk also touches upon the emerging threat of LLM indirect prompt injection. The importance of this topic is underscored by the unprecedented rate of AI adoption, exemplified by ChatGPT reaching 100 million users in just five days and the exponential growth of models on platforms like Hugging Face. As AI ushers in a new industrial revolution, securing these systems against malicious actors is paramount to ensuring a positive future, drawing parallels to the devastating impact of traditional software supply chain attacks like NotPetya and SolarWinds.

Background

▶ Watch: Overview of AI supply chain attack categories (0:55)

The rapid proliferation of AI technologies, especially Large Language Models (LLMs), has fundamentally redefined the technological landscape since the speakers' last appearance at BSidesSF. LLMs are now integrated into nearly every conceivable product and service, driving innovation and efficiency. This surge is further fueled by the rise of multimodal models, capable of understanding diverse data mediums, and the democratization of AI through open-source models that rival enterprise solutions, such as those leveraging Retrieval Augmented Generation (RAG). A tangible metric of this explosive growth is observed on Hugging Face, a platform akin to GitHub for machine learning. In just one year, the number of repositories on Hugging Face soared from 176,000 to 640,000, a trend that continues to accelerate minute by minute.

This unprecedented rate of adoption, while transformative, introduces significant security challenges. The speakers draw parallels to major traditional software supply chain attacks that have left "hard scars" on the industry, including NotPetya (2017), which caused billions in damages, SolarWinds, where the compromise of a build system led to widespread infiltration and even resulted in the Siso being charged by the SEC, Kaseya, Log4J, the Okta breach, and the recent XZ backdoor. These attacks share common denominators: the exploitation of trust between producer and consumer, and the reach of a compromised component, creating a one-to-many relationship that makes widely used platforms or products lucrative targets for adversaries.

The AI supply chain, while sharing these fundamental properties, introduces unique complexities. It is primarily composed of three distinct components: data, models, and tooling. Data refers to the input used to train models; models are the computational results of training on vast datasets, often deployed in applications; and tooling encompasses the frameworks, libraries, and services that enable the development, deployment, and monitoring of AI systems. The MLOps lifecycle, analogous to the DevOps lifecycle, involves stages like data collection, training, deployment, and monitoring. However, many existing serialization formats for models were developed before machine learning became prevalent and are inherently insecure by design. Furthermore, a significant challenge arises from vendors and developers often not classifying these insecurities as vulnerabilities, instead labeling them as "features" or shifting the blame to users for improper usage, as seen with Google's stance on TensorFlow or Python's warnings regarding Pickle. This lack of a unified security posture, coupled with the absence of standard security measures like digital signatures, integrity checks, and comprehensive malware scanning for large AI models, creates a fertile ground for sophisticated supply chain attacks.

Key Findings

▶ Watch: Malicious GitHub repos poisoning code-gen models (8:30)

The research presented by Eoin Wickens and Marta Janus uncovered critical vulnerabilities and attack vectors across the AI supply chain, categorizing them into data poisoning, model hijacking, and tooling exploitation, with an additional focus on indirect prompt injection.

Data Poisoning:

• Web-scale Dataset Poisoning: Research by Carini et al. demonstrated that by abusing data set formats that reference external URLs, attackers could buy up domains hosting images and poison 0.1% of a web-scale dataset for just $60. This seemingly small percentage was sufficient to significantly impact classifier performance.
• Real-world Examples: This concept extends to "PoisonGPT" and tools like Nightshade, which allows artists to poison their work, disrupting AI model training built upon it and causing label mismatches.
• Code Repository Poisoning: Millions of malicious repositories uploaded to GitHub pose a threat to code-generating LLMs. If models are trained on such compromised code, they will inevitably introduce vulnerabilities into enterprise applications, propagating the attack down the supply chain.
• RAG Poisoning: Retrieval Augmented Generation (RAG) systems, which use internal documents for LLM responses, are also susceptible. Attackers achieved a 90% attack success rate by injecting just five poison texts into a database containing millions of texts, highlighting the disproportionate impact of targeted poisoning.

Model Hijacking and Backdooring:

• Insecure Serialization Formats: Many machine learning frameworks rely on serialization formats that are "inherently insecure by design" or were not intended for large-scale ML models.
• Pickle: Python's serialization module allows for the loading of arbitrary Python modules and execution of functions. This "feature," not considered a vulnerability by Python, makes it trivial to inject malicious code (e.g., ransomware, reverse shells) into pre-trained models.
• Keras HDF5: While HDF5 itself is not inherently insecure for code execution, Keras allows the injection of "Lambda layers" for arbitrary expression execution. These layers are serialized using the Marshall module, which is as insecure as Pickle, enabling arbitrary code execution upon model loading.
• TensorFlow SavedModel: Based on Protobuf, SavedModel itself hasn't shown direct code execution vulnerabilities. However, TensorFlow provides functionalities to read/write files to disk, send/receive data over a network, and spawn additional processes. These capabilities can be combined to create malicious models that exfiltrate data, drop and execute binaries, or launch processes from within the TensorFlow context. Google maintains this is a "feature" and users should sandbox models.
• ONNX: Directory traversal vulnerabilities were discovered by the Hidden Layer team, allowing attackers to access files outside the intended directory. While some have been fixed, users are urged to update to the latest versions.
• R: A programming language specifically designed for statistical computing and machine learning, R was found to have a code execution vulnerability. This was one of the rare instances where the issue was taken seriously, a CVE was assigned, and a fix was implemented in the latest release.
• Neural Network Layer Injection: A more sophisticated attack involves injecting a layer of neurons into a model to subtly alter its behavior (e.g., a loan approval model approving all applications from individuals named "John"). This method is difficult to implement and detect, but the speakers anticipate future automated tools for such attacks.

Tooling Exploitation:

• Framework Vulnerabilities: ML frameworks are increasingly targeted.
• Shadow Ray: A vulnerability in the Ray framework (used for AI workloads, including by OpenAI) stems from a lack of authorization in its jobs API. Despite developers disputing it as a vulnerability, thousands of publicly exposed Ray servers have been found compromised.
• ClearML: Hidden Layer's team discovered multiple critical vulnerabilities, including code execution (via Pickle load), directory traversal, improper authentication, credential storage issues, CSRF, and XSS, which together enable a full attack chain.
• MLflow: Critical vulnerabilities have also been found in MLflow, another popular ML Ops tool.
• Dependency Compromise: Similar to traditional software, ML projects are vulnerable to compromised dependencies. The Torch-Trojan incident in December 2022 saw a malicious package uploaded to PyPI, acting as a Linux info stealer. This demonstrated the ease with which attackers could bypass dependency chain security.
• Package Confusion (LLM Hallucinations): LLMs can hallucinate non-existent software packages or libraries. Attackers can monitor these hallucinations, create malicious packages with the suggested names, and upload them to public repositories, tricking developers into downloading compromised dependencies.
• Hugging Face Safe Tensors Conversion Bot Compromise: Hidden Layer demonstrated that a malicious PyTorch model could compromise Hugging Face's service designed to convert insecure PyTorch models to the safer SafeTensors format. This allowed the exfiltration of the Hugging Face user token for the conversion bot account, enabling the impersonation of the legitimate service to send malicious pull requests to any model on the site, potentially replacing model weights entirely.

Indirect Prompt Injection:

• This novel attack vector involves injecting malicious prompts not directly into a chat window, but through external data sources like websites, audio, or images.
• Multimodal Prompt Injection: As described by Bagdasarian and Nassi at Black Hat, an AI-enabled chatbot browsing a website could be prompt-injected by a compromised image hosted on that site (e.g., Wikipedia). This raises questions about how to threat model for image-based adversarial examples in an AI-enabled world.

Technical Deep Dive

▶ Watch: TensorFlow 'features' abused for data exfiltration/malware (21:00)

The talk meticulously dissects the AI supply chain, focusing on its three core components—data, models, and tooling—and mapping attack vectors to key stages of the MLOps lifecycle: data collection, training, deployment, and monitoring.

The AI Supply Chain Components and MLOps Stages:

• Data: The raw material for AI, encompassing images, text, tabular data, or even references to external data sources like URLs or GitHub repositories. "Models are what they eat," meaning bad data leads to bad results.
• Models: The output of extensive computation over vast datasets, often bundled into applications or deployed on endpoints.
• Tooling: The software and services facilitating the development, deployment, and monitoring of AI systems.

The speakers illustrate these concepts through a realistic scenario: fine-tuning an LLM to create an internal programming co-pilot based on company-specific use cases.

Scenario: Internal Code Co-pilot LLM

1. Data Collection: Scraping 100,000 GitHub repositories and internal code repositories.

• Risks: Sourcing from untrusted or uncurated public repositories (e.g., random GitHub repos, subreddits) can introduce vulnerabilities. The model will "learn bad habits" if the dataset contains insecure code examples. Critical questions include: "Where did I source my data from?" and "Are there vulnerabilities in the code examples in the data set in the first place?"

1. Training: Fine-tuning an existing LLM model with the gathered data.

• Risks: The provenance of the base model is crucial. "Who built the model that I wanted to finetune?" "What was it trained on in the first place?" "Was it even built for code generation with for to be a production ready system?" An untrusted or improperly trained base model can propagate issues.

1. Deployment: Deploying the fine-tuned model internally across engineering teams.

• Risks: The model might produce erroneous output, specifically introducing vulnerabilities into generated code. It's essential to ask: "Has the model passed all possible evaluation checks?" and "Has it been evaluated by a security team?" The prevalence of "Shadow AI" (rapid, unvetted deployments) exacerbates this.

1. Monitoring: Observing the model introducing bugs and vulnerabilities into production systems.

• Risks: Lack of continuous monitoring post-deployment. Issues like "testing on the train data" can give false positives. Regular vulnerability assessments on generated code are critical to detect drift or malicious behavior.

Technical Deep Dive into Model Serialization Exploitation:

The core of model hijacking lies in exploiting the serialization formats used to store and exchange AI models.

• Pickle (Python): This Python module serializes Python objects to disk as binaries. A simple virtual machine interprets these binaries using around 70 instructions. Crucially, some instructions allow for the loading of arbitrary Python modules and the execution of Python functions with given arguments. This makes it "very easy to actually inject something into already pre-trained Model A simple python script that will... spawn a reverse shell or run any kind of malicious binary." The speakers demonstrated this with a simple "hello" script, noting that this capability is considered a "feature" by Python, not a vulnerability, despite warnings.
• HDF5 (Keras): The HDF5 file format is popular, especially with the Keras machine learning framework. While HDF5 itself doesn't inherently allow code execution, Keras facilitates the injection of "Lambda layers" into serialized models. These layers are designed for the execution of arbitrary expressions, which translates to arbitrary code execution. The serialization of these Lambda layers uses another Python module called Marshall, which is "just as insecure as pickle." The speakers confirmed they could inject arbitrary code into Keras HDF5 models, which executed upon loading.
• TensorFlow SavedModel: TensorFlow uses its own SavedModel format, based on Protobuf. The speakers stated that SavedModel itself, as a format, hasn't been found to have direct code execution vulnerabilities. However, TensorFlow as a framework provides powerful functionalities that can be abused:
• Reading and writing files to disk.
• Sending and receiving data over a network.
• Spawning additional processes.

By combining these "features," an attacker can create a malicious TensorFlow model that exfiltrates data, drops and schedules malicious binaries, or spawns processes from within the TensorFlow context. Google's response to these findings was that it's a feature, and models should be run in a sandbox.

• ONNX: The Open Neural Network Exchange (ONNX) format also had directory traversal vulnerabilities, allowing access to unauthorized file paths. While some were fixed, users are advised to use the latest versions.
• R: The R programming language, specifically designed for statistical computing and machine learning, was found to have a code execution vulnerability. This was a notable case where a CVE was assigned, and the developers released a fix, urging users to update.

Model Backdooring via Neural Network Layer Injection: Beyond serialization, attackers can inject a layer of neurons into a model to subtly alter its behavior. This is more complex to implement and detect, but it allows for targeted manipulation, such as modifying a loan approval model to approve all applications from individuals named "John."

Tooling and Dependency Exploitation:

• Ray Framework (Shadow Ray): The Ray framework, popular for AI workloads, suffered from the "Shadow Ray" vulnerability due to a lack of authorization in its jobs API. Despite developers arguing it's not a vulnerability if not exposed publicly, thousands of compromised Ray servers were found on the open internet.
• ClearML: Hidden Layer's team found a suite of vulnerabilities in ClearML, an open-source MLOps framework, including code execution (via Pickle), directory traversal, improper authentication, credential storage issues, and XSS/CSRF, enabling a full attack chain.
• Dependency Compromise (Torch-Trojan): The Torch-Trojan incident in December 2022 involved a malicious package uploaded to PyPI, which acted as a Linux info stealer. This highlighted the ease of exploiting the dependency chain.
• Package Confusion: LLMs can "hallucinate" non-existent package names. Attackers can then create and upload malicious packages with these hallucinated names to public repositories, tricking developers who follow LLM suggestions.

Hugging Face Safe Tensors Conversion Bot: This service was designed to convert insecure PyTorch models to the safer SafeTensors format. However, Hidden Layer demonstrated that a malicious PyTorch model could compromise the conversion service itself, allowing the exfiltration of the Hugging Face user token for the bot. This token could then be used to send malicious pull requests, impersonating the legitimate service, to any model on Hugging Face, potentially replacing model weights entirely.

Indirect Prompt Injection: This attack involves injecting prompts into an LLM not through direct user input, but through external data that the model processes. For example, a multimodal LLM browsing a website could be "prompt-injected" by an adversarial image embedded on that page, as demonstrated by Bagdasarian and Nassi. This opens up new threat modeling challenges for AI systems interacting with diverse, potentially untrusted, data sources.

Demo / Proof of Concept

▶ Watch: Shadow Ray vulnerability: disputed, thousands of compromised servers (30:00)

While the talk did not feature a live, in-depth demonstration, the speakers referenced several proof-of-concept (PoC) scenarios and past demonstrations to illustrate the feasibility and impact of the discussed attacks.

• Last Year's Ransomware Demo: Eoin Wickens reminded the audience of their previous BSidesSF talk where they "exploited the machine learning model pre trained resonant model we embedded some malicious code inside it and upon loading the model was executing ransomware." This served as a foundational example of model hijacking through serialization vulnerabilities, emphasizing the ease with which such attacks could be performed.
• Pickle Code Execution: For the Pickle serialization format, the speakers described and showed a simple Python script that, when injected into a model, would print "hello" to the console upon the model's loading. This illustrated the core mechanism of arbitrary code execution via Pickle's instruction set.
• Keras HDF5 Code Execution: Similarly, for Keras models serialized in HDF5 with Lambda layers, the speakers confirmed they were "able to inject arbitrary code in hdf five caras model and upon loading of the model the code was executed." This demonstrated the exploitability of the Marshall module used for Lambda layer serialization.
• TensorFlow Malicious Model Capabilities: While no direct code execution vulnerability was found in the TensorFlow SavedModel format itself, the speakers detailed how TensorFlow's built-in functionalities (file I/O, network communication, process spawning) could be combined to create a "malicious model that will either exfiltrate the data from the disk or drop a malicious binary schedule it for execution or spawn a process from inside the tensor flow context." This outlined the practical steps for building a malicious payload using legitimate framework features.
• Hugging Face Safe Tensors Conversion Bot Compromise: The Hidden Layer team successfully demonstrated that a malicious PyTorch model could compromise the Hugging Face conversion service. This PoC involved exfiltrating the Hugging Face user token for the bot account and subsequently sending malicious pull requests, impersonating the legitimate service, to other models on the platform. This proved the ability to "ultimately replace the weights entirely of any model across major organizations."

These examples, while not live code walkthroughs during the presentation, served as concrete evidence of the vulnerabilities discussed, reinforcing the practical implications of insecure practices within the AI supply chain.

Defensive Implications

▶ Watch: Hugging Face Safe Tensors conversion bot compromise (38:00)

Securing the AI supply chain requires a fundamental shift in mindset: treating machine learning models and frameworks with the same rigor as traditional software. The speakers outlined a comprehensive set of defensive strategies, ranging from basic hygiene to advanced industry initiatives.

General Principles:

• Apply Traditional Software Security: The most crucial takeaway is to apply all existing security measures from traditional software development (e.g., integrity checks, malware scanning, access control, regular updates, sandboxing) to ML models and frameworks. This forms the baseline upon which AI-specific solutions can be built.

Data Collection & Sourcing:

• Trusted Data Sources: Always source data from trusted, curated repositories. Avoid unverified public sources like random GitHub repositories or unmoderated forums.
• Data Curation & Validation: Critically examine data for vulnerabilities or malicious content. Models mirror their input; "garbage in, garbage out."
• Vulnerability Assessment: Perform vulnerability assessments on code examples or any other data used for training, especially for code-generating models.

Model Sourcing, Training & Deployment:

• Provenance Verification: Thoroughly investigate the origin of any pre-trained model. "Who built the model that I wanted to finetune?" "What was it trained on?" "Was it built for a production-ready system?"
• Integrity Checks: Always verify the integrity of downloaded models using checksums provided by publishers.
• Malware Scanning: Scan downloaded models for malware. While many traditional anti-malware products may not yet fully support ML models, this is an evolving area.
• Sandbox Execution: Crucially, run any untrusted or newly acquired model in a sandbox environment first to observe its behavior and detect any unexpected or malicious actions before deployment.
• Rigorous Evaluation: Subject models to comprehensive evaluation checks, including security assessments by dedicated security teams, to identify erroneous outputs or vulnerabilities. Avoid "Shadow AI" deployments that bypass security scrutiny.
• Access Control: Implement strict access control policies for model repositories and deployment environments (e.g., S3 buckets).

MLOps Tooling & Dependencies:

• Regular Updates: Keep all MLOps software, machine learning frameworks, and libraries regularly updated to patch known vulnerabilities (e.g., ONNX, R).
• Proper Authentication: Ensure all frameworks and tools have robust authentication mechanisms, especially if exposed to external networks. Do not rely on developers' assumptions that tools will only be used locally (e.g., Ray framework).
• Dependency Management: Be vigilant about third-party dependencies. Verify the authenticity of packages and be wary of package confusion attacks stemming from LLM hallucinations.

Monitoring:

• Continuous Monitoring: Implement continuous monitoring of model performance and behavior post-deployment. Look for signs of drift, unexpected outputs, or anomalous activity.
• Vulnerability Assessment on Generated Output: For generative AI, continuously perform vulnerability assessments on the code or content generated by the model.
• Attack Detection: If models are exposed externally, implement checks to detect inference attacks or model theft attempts.

Industry-Wide Initiatives & Best Practices:

The industry is actively developing frameworks and standards to address AI security:

• MITRE ATLAS: A knowledge base of tactics, techniques, and case studies for adversarial AI, offering mitigations for AI-specific attacks.
• OWASP: Provides resources like the Top 10 risks for Machine Learning, Top 10 for LLMs, and the OWASP AI Exchange.
• NIST AI Risk Management Framework: One of the pioneering frameworks for managing AI risks.
• Regulatory Frameworks: Initiatives like the EU Artificial Intelligence Act and ISO standard 42001 are establishing regulatory and standardization guidelines.
• AI Bill of Materials (AI BOM / ML BOM): Analogous to software BOMs, these aim to provide transparency into the components of AI systems.
• AI Security Posture: Efforts to adapt and port traditional software security posture management to the AI domain.
• Model Signing: Development of digital signatures for machine learning models to prove integrity, provenance, and detect tampering.
• Lineage Tracking: Tracking changes and versions of machine learning models, crucial for fine-tuned LLMs.

Positive Industry Changes Noted:

• Keras: Now defaults to not allowing deserialization of Lambda layers to prevent code execution.
• Hugging Face: Fixed the conversion bot vulnerability, is implementing rudimentary malware scanning for models, and is enhancing user/repository verification.
• R: The code execution vulnerability has been patched in the latest release.

These defensive measures, when implemented comprehensively, can significantly bolster the security posture of AI systems against the multifaceted threats emerging in the supply chain.

Key Takeaways

• Multifaceted Attack Surface: The AI supply chain presents numerous points of compromise, including data poisoning during collection, hijacked or backdoored models during sourcing, vulnerabilities in MLOps tooling and compromised packages, and emerging threats like indirect prompt injection and prediction tempering.
• Treat AI as Software: The most fundamental defense is to apply all existing security measures and best practices from traditional software development (e.g., integrity checks, malware scanning, access control, regular updates, sandboxing) to machine learning models and frameworks.
• Verify Provenance and Integrity: Rigorously verify the origin, integrity, and trustworthiness of all AI supply chain components, including training data, pre-trained models, and third-party tools. Utilize checksums, verify publishers, and scrutinize data sources.
• Continuous Monitoring is Essential: Post-deployment, AI systems require continuous monitoring for performance drift, unexpected behavior, and the introduction of vulnerabilities into generated outputs. Sandbox untrusted models before integration.
• Beware of "Features": Many vulnerabilities in ML frameworks are not classified as such by vendors but rather as "features" (e.g., Pickle's arbitrary code execution, TensorFlow's file I/O capabilities). Defenders must understand these inherent risks and implement compensating controls.
• Proactive Security is Paramount: Given the explosive growth and integration of AI, proactive security measures, robust threat modeling, and engagement with evolving industry standards (e.g., MITRE ATLAS, AI BOM, model signing) are critical to prevent future widespread compromises.

About the Speaker(s)

Eoin Wickens and Marta Janus are researchers at Hidden Layer, where they are part of the Synaptic Adversarial Intelligence (SAI) team. Both have been actively researching adversarial machine learning and AI security for several years. They previously presented at BSidesSF, with their 2023 talk titled "Sleeping with one AI open," which introduced attacks on AI and machine learning. Their work focuses on understanding and mitigating the security risks associated with the rapidly evolving AI landscape.