AIBOM: Powering Transparency and Security in AI and Software Supply Chains

AIBOM: Powering Transparency and Security in AI and Software Supply Chains

Speakers: Dimiter Raidman, CTO & Co-founder, SyzItz Cab; Helen Oakley, SAP; Daniel, Manifest Cyber

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=c9bfmYzorhQ

Overview

In an era where Artificial Intelligence (AI) rapidly integrates into critical business operations, the security of AI models and their underlying supply chains has become paramount. This talk by Dimiter Raidman, CTO and co-founder of SyzItz Cab, delves into the critical need for AI Bill of Materials (AIBOMs) to ensure transparency, security, and compliance within the burgeoning AI landscape. Raidman, who also co-leads the CISA Tiger Team defining AI SBOMs alongside Helen Oakley from SAP and Daniel from Manifest Cyber, highlights the unique risks and vulnerabilities inherent in AI systems that traditional software supply chain security measures fail to address adequately.

The session opens with a compelling hypothetical scenario set in 2030, where Acme Investment Portfolio Trading suffers an $80 million loss due to an AI model making overly aggressive and incorrect market recommendations. The root cause: a sophisticated adversary exploiting a vulnerability in the AI model training pipeline of an open-source model, injecting poisoned training data. This incident, inspired by real-world events in the housing market, underscores the severe financial and reputational consequences when organizations lack visibility into their AI models' provenance and integrity. AIBOMs are presented as the essential solution to understand and mitigate such risks, providing the necessary auditability and transparency to navigate the complex world of AI governance, security, and compliance.

Background

▶ Watch: Introduction to AI as-bombs and emerging risks (0:00)

The modern machine learning (ML) software supply chain is a complex ecosystem. It typically involves ML developers and data scientists who either train models from scratch or, more commonly, fine-tune base models acquired from open-source repositories like Hugging Face. These models are then built, stored in model registries, and subsequently downloaded for deployment on inference servers, often leveraging specialized hardware like Nvidia GPUs. Throughout this pipeline, numerous components—data sets, models, and frameworks—are frequently sourced from open-source communities. This reliance on external components introduces a significant "danger zone," as vulnerabilities or compromises within any part of this chain can have cascading effects, for which the adopting organization ultimately bears responsibility.

Current challenges in AI adoption are multifaceted. Organizations are operating with nascent MLOps practices, still figuring out how to properly secure these pipelines, unlike the more mature DevSecOps processes for traditional software. AI systems are inherently high complexity, with intricate layers, diverse data models, and infrastructure distinct from conventional software. This complexity is further exacerbated by the pressure to accelerate time-to-market and adopt advanced agentic AI, which can increase system complexity by hundreds of times. Regulatory bodies are also increasingly scrutinizing AI, demanding better risk containment and adherence to responsible practices. Compounding these issues are a lack of adequate training for developers in building secure AI systems and the continuously evolving nature of models, leading to model drift when encountering unexpected data.

This environment creates fertile ground for a range of AI-specific vulnerabilities and risks. Beyond traditional software dependencies, AI systems face threats such as vulnerable dependencies spanning models, datasets, and training frameworks. Data poisoning stands out as a primary risk, where malicious or manipulated data is injected into training sets to subtly alter model behavior and produce undesirable outcomes. Similarly, model poisoning involves embedding Trojans or backdoors directly into models, enabling malicious execution during inference. Other critical threats include Denial of Service (DoS) attacks that can cripple AI services, model inversion attacks aiming to extract sensitive training data (e.g., PII, financial data) embedded within a model, and prompt injections. The latter, often seen in large language models, allows adversaries to "social engineer" the AI, manipulating its responses or actions, potentially leading to unauthorized data access or control over tools the AI uses. For example, the speaker recounted an instance where a chatbot was manipulated to use an offensive word by exploiting its perceived "sensitivity."

The speaker also highlighted that vulnerabilities extend beyond software and data to hardware and underlying frameworks. As an example, a high-severity vulnerability in Nvidia's framework for running models, affecting all Nvidia Jetson hardware with their Linux distribution, was patched last year and documented on NVD. More alarmingly, a vulnerability in ZenML, a widely used MLOps platform, allowed unauthorized access to AI building and deployment pipelines through an API endpoint. Despite its critical nature, this ZenML vulnerability, discovered over a year ago, lacked any corresponding CVE mappings on the NVD website, illustrating a significant gap in current vulnerability tracking for AI-specific components and platforms. This lack of transparency and standardized vulnerability disclosure for AI components underscores the urgent need for a systematic approach like AIBOMs.

Key Findings

▶ Watch: Why AI as-bombs are crucial for tracing incidents (2:18)

The central finding of this talk is the indispensable role of AIBOM (Artificial Intelligence Bill of Materials) as a foundational tool for ensuring transparency, security, and compliance in the rapidly evolving AI landscape. Dimiter Raidman emphasizes that AIBOM is not a radical departure but rather an extension of the familiar Software Bill of Materials (SBOM) concept, specifically tailored to encompass the unique components and complexities of AI systems.

Crucially, the good news is that established SBOM standards, CycloneDX and SPDX, are already equipped to support AIBOMs. Specifically, CycloneDX versions 1.5 and above (with 1.6 being recommended) and SPDX versions 3.0 and above (with 3.0.1 recommended) include the necessary structures to represent AI components. This existing compatibility significantly lowers the barrier to adoption, allowing organizations to leverage familiar tooling and processes for AI asset management.

By providing a detailed inventory of all constituents within an AI system—from models and datasets to frameworks and hardware—AIBOMs deliver essential trust, compliance, and innovation. They enable organizations to understand the provenance of their AI assets, identify potential risks, and meet increasing regulatory demands. The CISA Tiger Team, in which Raidman, Oakley, and Daniel are actively involved, has identified seven key use cases for AIBOMs, further solidifying their practical utility across various aspects of the AI lifecycle. These use cases, currently undergoing review with CISA, are expected to be published shortly, providing a roadmap for AIBOM implementation and utility.

Technical Deep Dive

▶ Watch: Understanding the modern ML software supply chain and its risks (3:40)

An AIBOM, while building upon the principles of an SBOM, necessitates the inclusion of specific metadata and component types unique to AI systems. The speaker meticulously outlined these critical elements that differentiate an AIBOM and provide the granular transparency required for AI security.

At its core, an AIBOM must detail the model source and architecture. This includes identifying the foundational models used, such as Rakuten's utilization of Mal Kajel ULM, and understanding the underlying structure upon which the AI was built. Unlike traditional software, model versioning in AI presents a significant challenge. Conventional semantic versioning (e.g., OpenSSL 1.1.1L) is largely absent in the AI world. Instead, models like "Rakuten 7B" might represent a general version with numerous underlying commits, each potentially using different data sources without their own versioning. To achieve precise identification and traceability, AIBOMs must rely on commit hashes of model changes and associated data sets, providing an immutable record of the model's evolution.

Beyond provenance, model performance metrics are a critical inclusion. These metrics, such as bias, precision, error rates, and confidence levels for inferences, are vital for assessing a model's quality, fairness, and potential for unintended consequences. Just as a developer might check the download count or maintainer reputation of an NPM package, organizations need to perform due diligence on open-source AI models by scrutinizing these performance indicators to ensure they align with desired operational parameters and ethical considerations.

The AIBOM also necessitates a clear enumeration of model dependencies. This includes not just the constituent AI models but also the tools and frameworks used for their creation, operation, and fine-tuning. Examples include popular libraries like PyTorch and Transformers, which are integral to the model's functionality and can introduce their own vulnerabilities.

Crucially, model licensing information extends beyond the model itself to encompass the licensing of the data sets used for training. This is a significant departure from traditional software, as data sets can contain sensitive, private, or confidential information, and their licensing dictates usage rights. Organizations must understand who owns the data, what intellectual property rights are involved, and whether its use infringes on any agreements, especially when model creators may not fully disclose their data sources due to proprietary concerns. This granular understanding is further enhanced by including classification data for these data sets, categorizing them by sensitivity (e.g., sensitive, confidential, private) to inform risk assessment and data handling policies.

Integrating AIBOMs into ML SecOps pipelines is depicted as a multi-stage process. It begins with the generation of third-party models, followed by an organization's internal development, training, and fine-tuning phases, where proprietary data and modifications are introduced. The resulting model, now a hybrid of third-party and proprietary components, is then deployed to production. The AIBOM acts as a continuous record, evolving with each stage to reflect the complete composition.

A significant challenge highlighted in the talk is the current lack of readily available, automated AIBOM generators for AI models. Unlike source code or binary analysis tools for SBOMs, the complexity and unstructured nature of AI assets make automated extraction difficult. Current approaches involve integrating with APIs from platforms like Hugging Face, enriching AIBOMs from existing model metadata, performing dependency analysis to infer data set reliance, and adding custom fields to capture unique attributes. A major hurdle in this process is the prevalence of model cards as free-form text. While model cards contain valuable information, their unstructured nature makes it incredibly complex to programmatically extract and map data to the standardized attributes required by CycloneDX or SPDX.

However, a promising solution is on the horizon: Helen Oakley has developed an AIBOM generator that she plans to release as open-source. This tool, to be announced at the upcoming RSA AIBOM workshop, will address the current generation gap by allowing users to provide a URL and automatically generate an AIBOM for the specified AI model. This innovation is expected to significantly streamline the process of creating AIBOMs for third-party models, enabling organizations to integrate them into their broader infrastructure and application SBOMs.

Demo / Proof of Concept

▶ Watch: Key challenges organizations face in secure AI adoption (4:47)

While the talk provided a comprehensive conceptual and technical overview of AIBOMs, it did not include a live demonstration or proof of concept of an AIBOM in action. The speaker, Dimiter Raidman, instead announced an upcoming development by his CISA Tiger Team co-lead, Helen Oakley. She has developed an AIBOM generator that will be released as open-source and publicly announced at the AIBOM workshop at RSA. This tool is designed to take a URL as input and automatically create an AIBOM for the specified AI model, addressing a critical current gap in automated AIBOM generation.

Defensive Implications

▶ Watch: Common AI vulnerabilities: data poisoning, trojans, DoS attacks (8:10)

The insights presented in this talk offer crucial guidance for defenders navigating the complex and rapidly evolving AI threat landscape. The overarching defensive implication is the imperative to proactively adopt and integrate AIBOMs as a foundational element of an organization's AI security strategy.

Firstly, organizations must treat AIBOMs as their primary inventory and transparency tool for AI components. This means generating AIBOMs not only for internally developed models but critically, for all third-party AI models and their constituent datasets and frameworks. By doing so, defenders gain unprecedented visibility into their AI supply chain, understanding the provenance, dependencies, and potential vulnerabilities of every component.

With AIBOMs in hand, organizations can then perform rigorous risk assessments on their AI models. Leveraging new frameworks from organizations like OWASP, security teams can analyze the identified components for known vulnerabilities (like the Nvidia Jetson or ZenML examples), assess the integrity of training data, evaluate model performance metrics for bias or anomalies, and scrutinize licensing information to prevent legal entanglements. This proactive assessment allows for informed decision-making regarding model adoption and deployment.

Continuous monitoring is another critical defensive measure. AIBOMs enable ongoing integrity checks and anomaly detection across both development and production workloads. If a vulnerability is discovered in an upstream component (e.g., a specific PyTorch version or a base model from Hugging Face), the AIBOM provides the necessary data to quickly identify all affected internal systems, facilitating rapid patching and remediation before harm is done. This significantly reduces the window of exposure to newly discovered threats.

Furthermore, AIBOMs are instrumental in supporting adherence to regulations. As AI governance frameworks mature globally, the ability to collect and present evidence of due diligence, risk assessment, and control over AI components will become non-negotiable. AIBOMs provide the auditable records necessary to demonstrate compliance and responsible AI practices.

Beyond technical measures, defenders must also prioritize training and education. Just as employees are trained against phishing, developers and data scientists need to be educated on how to build secure and resilient AI systems, understanding the implications of open-source dependencies, data provenance, and secure coding practices for AI. The importance of sanitizing and properly treating data at every stage of the ML pipeline cannot be overstated, as data poisoning remains a top threat.

Finally, the talk issues a direct call to action: engage with the AIBOM Tiger Team (the CISA-led initiative). By participating in this community, organizations can contribute to the development of standards, share best practices, and collectively strengthen the security posture of the entire AI ecosystem.

Key Takeaways

• AI systems introduce unique and complex supply chain risks that extend beyond traditional software, encompassing models, training data, frameworks, and even hardware. Vulnerabilities like data poisoning, model inversion, and prompt injection require specialized transparency.
• AIBOMs (Artificial Intelligence Bill of Materials) are essential for AI transparency, security, and governance. They provide a critical inventory of all components within an AI system, enabling organizations to understand provenance and manage risks effectively.
• Existing SBOM standards (CycloneDX 1.5+ and SPDX 3.0+) already support AIBOMs, facilitating adoption, but AI-specific metadata is crucial. This includes precise versioning via commit hashes, model performance metrics (bias, precision), and comprehensive licensing details for both models and their training datasets.
• Automated AIBOM generation for AI models is an emerging necessity. While currently challenging due to unstructured model cards, tools like Helen Oakley's upcoming open-source generator will significantly streamline the process, enabling better integration into ML SecOps pipelines.
• Defenders must proactively leverage AIBOMs for risk assessment, continuous monitoring, and compliance. This enables faster identification and patching of vulnerabilities, ensures data integrity, and provides auditable evidence for regulatory adherence.
• Engagement with community efforts like the CISA AIBOM Tiger Team is vital for contributing to and benefiting from the development of AIBOM standards and best practices, collectively enhancing AI supply chain security.

About the Speaker(s)

Dimiter Raidman is the CTO and co-founder of SyzItz Cab, a company dedicated to providing solutions for software supply chain security and SBOM management. In addition to his role at SyzItz Cab, Dimiter is a key participant in the CISA Tiger Team, where he co-leads the initiative to define AI SBOMs. His expertise lies in understanding and mitigating the evolving risks within both traditional software and emerging AI supply chains.

Helen Oakley from SAP and Daniel from Manifest Cyber are also instrumental in the CISA Tiger Team, co-leading the effort to define AI SBOMs alongside Dimiter Raidman. Helen Oakley is notably developing an open-source AIBOM generator, a tool poised to significantly advance the practical implementation of AI Bill of Materials.