Threat Modeling in the Age of AI

Threat Modeling in the Age of AI

Speakers: Adam Shostack

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=tYOJeChUM4M

Overview

In an insightful presentation at DEF CON 32’s AppSec Village, renowned threat modeling expert Adam Shostack addressed the critical intersection of artificial intelligence and cybersecurity. His talk, "Threat Modeling in the Age of AI," provided a foundational perspective on how established security practices can and must adapt to the rapidly evolving landscape dominated by large language models (LLMs). Shostack's core message underscored the enduring relevance of proactive security design, likening threat modeling to the crucial "measure twice, cut once" principle in physical construction, now applied to the intricate architecture of AI systems.

The presentation aimed to demystify threat modeling for those new to the discipline, while simultaneously demonstrating its applicability to the cutting-edge challenges posed by AI. Shostack highlighted a dual approach: applying traditional threat modeling techniques to AI systems to identify vulnerabilities and risks, and conversely, leveraging AI and LLMs as tools to enhance the security of other systems. This nuanced perspective moves beyond the sensationalized fears of sentient AI, grounding the discussion in practical, actionable security engineering principles relevant to today's prevalent LLM deployments.

Shostack's contribution to the DEF CON community is particularly significant given his extensive background and influence in the field of threat modeling. His talk served as a call to action for security professionals and developers alike, urging them to integrate security thinking early in the AI development lifecycle. By doing so, organizations can anticipate and mitigate potential security flaws, thereby saving invaluable human time, energy, and resources that would otherwise be expended on costly late-stage remediation or, worse, responding to breaches.

Background

▶ Watch: Introduction and talk's main topic: AI threat modeling (0:00)

The concept of artificial intelligence has long captivated the public imagination, often conjuring dystopian visions of rogue machines like the Terminator or the malevolent Hal 9000 from "2001: A Space Odyssey." These cinematic portrayals painted a picture of sentient, existential threats that, while compelling, largely diverged from the practical realities of AI development. As Adam Shostack humorously noted, the "age of AI" has arrived, but it manifests not as menacing robots, but as "way less threatening looking, way less cinematic" systems, primarily large language models (LLMs) that power everything from chatbots to sophisticated data analysis tools. This shift from theoretical, existential threats to tangible, everyday software systems necessitates a re-evaluation of how we approach their security.

The problem, as Shostack articulates, is that while the nature of AI has evolved, the fundamental challenges of securing complex software systems remain. Historically, software development has often prioritized functionality and speed over security, leading to a reactive posture where vulnerabilities are discovered and patched post-deployment. This approach is not only inefficient but also costly. Shostack draws a powerful analogy from physical construction: "measure twice, cut once." In carpentry or construction, meticulous planning and measurement precede any cutting, preventing wasted material and effort. In software, the "material" wasted by neglecting early security considerations is "human time and energy." When security is an afterthought, teams often find themselves in "crazy arguments" about shipping deadlines versus critical fixes, or worse, needing to completely rewrite sections of code because fundamental security flaws were embedded from the start.

This context underscores why threat modeling is not just a best practice but a critical discipline. Threat modeling, at its core, is "using models to help us think about security." It involves "abstracting away some details so we can see the forest rather than the trees or the twigs." By creating simplified representations of a system, security professionals can identify potential vulnerabilities, attack vectors, and threat actors before code is even written or deployed. Shostack, a leading authority in this field, emphasizes that threat modeling allows teams to "anticipate these problems, we can find these problems when there is time to do something about them." The advent of AI, with its novel architectures, data dependencies, and interaction paradigms, only amplifies the urgency and necessity of this proactive security approach. Without it, the "age of AI" risks becoming an age of unprecedented security vulnerabilities and wasted development resources.

Key Findings

▶ Watch: Talk agenda: threat modeling overview, AI, and LLM security (2:00)

Adam Shostack's talk distilled several crucial insights regarding the application of threat modeling in the contemporary AI landscape. The overarching finding is that while the technology may be new, the fundamental principles of proactive security remain indispensable, and indeed, become even more critical.

Firstly, Shostack firmly established that threat modeling is the "measure twice cut once of cyber security." This analogy highlights the profound economic and operational benefits of integrating security considerations early in the development lifecycle. By proactively identifying potential security weaknesses and attack paths, organizations can prevent costly rework, avoid contentious debates over shipping schedules versus security fixes, and ultimately deliver more robust and trustworthy systems. This finding emphasizes that the cost of fixing a security flaw increases exponentially the later it is discovered in the development process.

Secondly, the talk identified a dual application of threat modeling in the age of AI. This is a pivotal contribution, moving beyond a monolithic view of AI security:

1. **Applying threat modeling techniques to artificial intelligence systems, particularly LLMs:** This involves understanding the unique architecture, data flows, and interaction models of AI systems to systematically identify their specific vulnerabilities. This means adapting existing threat modeling frameworks to analyze training data integrity, model inference security, prompt injection risks, data privacy within LLMs, and potential misuse scenarios.
2. **Using LLMs to secure systems:** This perspective recognizes AI not just as a target of threats but also as a powerful tool for defense. While Shostack noted he would touch on this briefly due to a more in-depth preceding talk, the mere inclusion of this point highlights the potential for AI-driven security analysis, automated vulnerability detection, and intelligent threat response, thereby leveraging AI's capabilities to enhance overall cybersecurity posture.

Finally, Shostack underscored that threat modeling enables anticipating problems and finding them when there is time to do something about them. This finding stresses the proactive nature of the discipline. Instead of reacting to breaches or post-deployment vulnerability reports, threat modeling empowers teams to design security in from the ground up. This foresight is particularly vital in the rapidly evolving and often opaque world of AI, where novel attack vectors can emerge quickly, and the impact of a breach could be far-reaching, encompassing data compromise, intellectual property theft, or even system manipulation.

Technical Deep Dive

▶ Watch: Threat modeling: the 'measure twice cut once' of cybersecurity (3:20)

The technical deep dive into threat modeling in the age of AI, as presented by Adam Shostack, centers less on specific AI-related attack vectors (which often evolve rapidly) and more on the foundational methodology and mindset required to secure these complex systems. Shostack's definition of threat modeling—"using models to help us think about security" and "abstracting away some details so we can see the forest rather than the trees or the twigs"—provides the cornerstone for this approach.

The essence of threat modeling lies in its capacity to simplify complexity. AI systems, especially large language models (LLMs), are inherently complex. They involve vast datasets for training, intricate neural network architectures, sophisticated inference mechanisms, and diverse deployment scenarios (cloud, on-premise, edge). Attempting to secure such systems without a structured approach is akin to navigating a dense forest without a map. Threat modeling provides that map by encouraging the creation of abstract models that represent the system's components, data flows, trust boundaries, and interactions.

When applying threat modeling to LLMs, the core techniques remain consistent with established methodologies (such as STRIDE, Data Flow Diagrams (DFDs), or Attack Trees), but the specific elements being modeled and the types of threats considered must be adapted. A threat modeler would begin by defining the scope of the LLM system:

• Data Sources: What data is used for training and fine-tuning? What are its origins, integrity, and privacy implications?
• Model Architecture: How is the LLM constructed? Where are its components deployed?
• Input/Output Mechanisms: How do users interact with the LLM? What are the interfaces, APIs, and data formats?
• Deployment Environment: Where does the LLM run? What are the surrounding infrastructure components (databases, APIs, user authentication systems)?

By mapping these elements, security professionals can identify potential areas of concern. For an LLM, this might involve:

• Input Validation: Are prompts sanitized to prevent prompt injection or jailbreaking?
• Data Poisoning: Can an adversary inject malicious data into the training set to manipulate the model's behavior or introduce backdoors?
• Model Evasion: Can an attacker craft inputs that cause the model to produce undesirable or harmful outputs while appearing benign?
• Data Exfiltration: Can an LLM inadvertently or maliciously leak sensitive information from its training data or internal state?
• Access Control: Who has access to the LLM, its training data, and its fine-tuning capabilities?
• Integrity of Output: Can the LLM's outputs be tampered with or misrepresented?

The "abstraction" aspect of threat modeling is crucial here. Instead of getting lost in the millions or billions of parameters of an LLM, the modeler focuses on the high-level interactions and trust boundaries. For example, a data flow diagram for an LLM might depict user input flowing into a front-end application, which then communicates with an LLM inference service, which in turn might query a knowledge base, before returning an output to the user. Each arrow and component in this diagram becomes a point for security analysis, prompting questions about potential threats (e.g., spoofing the user, tampering with the prompt, information disclosure from the knowledge base, denial of service on the inference service).

Furthermore, Shostack briefly touched upon the inverse application: using LLMs to secure systems. While not elaborated upon in this specific segment, this concept implies leveraging AI's analytical capabilities for security tasks. For instance, LLMs could be trained on vast datasets of vulnerability reports, threat intelligence feeds, or code repositories to:

• Automate vulnerability detection: Identifying common coding patterns that lead to security flaws.
• Enhance threat intelligence: Processing and correlating vast amounts of security data to uncover emerging threats.
• Assist in incident response: Quickly analyzing logs and alerts to identify attack patterns and suggest remediation steps.
• Generate secure code suggestions: Offering real-time feedback to developers on more secure coding practices.

This dual perspective highlights the dynamic relationship between AI and cybersecurity. AI systems present new attack surfaces that require diligent threat modeling, but they also offer powerful new tools that can augment human defenders. The challenge lies in applying established security engineering principles with an adaptive mindset, recognizing the unique characteristics and evolving nature of AI technology. By systematically modeling these systems, security professionals can move beyond reactive patching to proactive, design-time security, ensuring that the "forest" of AI innovation is secure from the ground up.

Demo / Proof of Concept

▶ Watch: Importance of threat modeling: preventing wasted human time (3:40)

The provided transcript segment for Adam Shostack's talk, "Threat Modeling in the Age of AI," does not include any demonstration or proof of concept. The speaker was stepping in at the last minute and focused on delivering the core conceptual framework and the importance of threat modeling, particularly its application to AI systems, rather than showcasing specific technical implementations or tools.

Defensive Implications

▶ Watch: Call to action: apply threat modeling to all systems, including AI (4:10)

The defensive implications of Adam Shostack's insights on threat modeling in the age of AI are profound and directly actionable for security professionals, developers, and organizations. The core message empowers defenders to shift from a reactive stance to a proactive, preventative security posture, especially crucial given the nascent and rapidly evolving nature of AI technologies.

Firstly, defenders must prioritize and integrate threat modeling early in the AI development lifecycle. Shostack's "measure twice, cut once" analogy is a direct call to action. Organizations developing or deploying AI systems, particularly LLMs, should embed threat modeling as a mandatory step at the design phase, before significant code is written. This ensures that potential vulnerabilities related to data poisoning, prompt injection, model integrity, privacy, and system misuse are identified and addressed architecturally, rather than attempting to patch them later. Early identification dramatically reduces the cost and complexity of remediation, preventing the "crazy arguments" and wasted "human time and energy" that arise from late-stage security findings.

Secondly, security teams need to adapt existing threat modeling methodologies to the unique characteristics of AI systems. While the fundamental principles of identifying assets, threats, vulnerabilities, and mitigations remain, their application to LLMs requires a nuanced understanding of AI-specific components. Defenders should consider:

• Data Supply Chain Security: Ensuring the integrity and confidentiality of training data, fine-tuning data, and input prompts. This includes scrutinizing data sources, access controls, and transformation processes.
• Model Integrity and Confidentiality: Protecting the LLM itself from unauthorized access, tampering, or intellectual property theft. This involves securing the model weights, architectures, and inference endpoints.
• Interaction Security: Analyzing the interfaces through which users and other systems interact with the LLM. This is where threats like prompt injection, data leakage via output, and denial of service attacks are particularly relevant.
• Adversarial AI Techniques: Understanding and anticipating how malicious actors might try to manipulate, evade, or exploit the AI system's learning and decision-making processes.

Thirdly, defenders should explore and leverage AI as a tool for enhancing security. While Shostack's talk only briefly touched on this, the implication is clear: AI can become an invaluable ally in the fight against cyber threats. Security teams should investigate how LLMs and other AI technologies can be utilized for:

• Automated Threat Intelligence: Processing vast amounts of data to identify emerging attack patterns, malware signatures, and threat actor tactics.
• Vulnerability Management: Assisting in code analysis, identifying common security flaws, and suggesting remediation.
• Incident Response: Accelerating the analysis of security logs, correlating events, and providing rapid insights during a breach.
• Security Orchestration, Automation, and Response (SOAR): Enhancing the efficiency and effectiveness of security operations.

In essence, the defensive implications revolve around proactive engagement, continuous adaptation, and strategic utilization of technology. By embracing threat modeling as a cornerstone of AI system development and by intelligently deploying AI to bolster their own defenses, organizations can build more resilient, trustworthy, and secure AI-powered futures, mitigating the risks inherent in this transformative technology.

Key Takeaways

• Threat modeling is paramount for AI security: It provides a structured, proactive approach to identify and mitigate risks in complex AI systems, particularly LLMs.
• "Measure twice, cut once" applies directly to cybersecurity: Integrating security considerations early in the AI development lifecycle saves significant time, energy, and resources compared to reactive patching.
• AI systems present unique threat surfaces: Traditional security practices must be adapted to address novel vulnerabilities related to data poisoning, prompt injection, model evasion, and intellectual property theft in LLMs.
• Threat modeling helps anticipate problems: By creating abstract models of AI systems, security professionals can identify potential flaws and attack vectors before they are exploited.
• AI can be a powerful tool for defense: Beyond securing AI systems, LLMs and other AI technologies hold significant potential to enhance existing security operations, from threat intelligence to incident response.
• Proactive security is cost-effective: Addressing security concerns at the design stage of AI development is far more efficient and less disruptive than fixing issues post-deployment.

About the Speaker(s)

Adam Shostack is a highly respected and influential figure in the field of cybersecurity, particularly renowned for his extensive work on threat modeling. As indicated in his talk, he has dedicated considerable time and energy to developing and advocating for effective threat modeling practices. His expertise lies in helping organizations think systematically about security, abstracting complex systems into manageable models to identify vulnerabilities early in the development process. His contributions have significantly shaped how many professionals approach security design and risk assessment.