On Your Ocean's 11 Team, I'm the AI Guy (technically Girl)

On Your Ocean's 11 Team, I'm the AI Guy (technically Girl)

Speakers: Harriet Farlow

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=pTSEViCwAig

Overview

In "On Your Ocean's 11 Team, I'm the AI Guy (technically Girl)," Harriet Farlow, a seasoned expert in AI security, delves into the critical vulnerabilities emerging as organizations rapidly adopt artificial intelligence. Delivered at DEF CON 32, the talk uses the high-stakes environment of a casino as a compelling analogy to illustrate the inherent insecurities of modern AI systems and the severe financial and reputational risks they pose. Farlow's presentation aims to bridge the knowledge gap between traditional cybersecurity professionals and the burgeoning field of AI security, demonstrating how machine learning models, particularly facial recognition AI, can be exploited.

Farlow's core objective is to challenge the perception that AI systems are inherently robust or secure by design. Through the lens of "hacking the casino," she highlights that while organizations are increasingly integrating AI into critical operations—from surveillance to customer analytics—the security implications are often overlooked until significant financial loss or brand damage occurs. The talk serves as a stark warning, emphasizing that the current rapid deployment of AI often outpaces the implementation of adequate security measures, leaving these systems ripe for exploitation by sophisticated adversaries.

This presentation is particularly salient given the current "inflection point" of AI adoption across virtually all industries. Farlow's work, including her collaboration with Casino Canberra, underscores a practical, real-world approach to identifying and addressing these vulnerabilities. By focusing on a tangible and relatable target like a casino, the talk effectively communicates the urgency and impact of AI security threats to a broad audience, encouraging both AI developers and cybersecurity practitioners to proactively secure these increasingly vital technologies.

Background

▶ Watch: Speaker introduction, Australian roots, and the 'shoey' tradition (0:00)

Harriet Farlow's journey into AI security spans over a decade, originating from an undergraduate background in physics that, like many, led her into data science. Her career trajectory includes stints as a data scientist in defense projects in Australia, working at a startup in the United States, and contributing to the Australian government. It was during her PhD in machine learning security, which she undertook amidst the COVID-19 pandemic, that she identified a significant void: a lack of widespread discussion and focus on machine learning security in 2021. This realization became the catalyst for her to establish Mileva Security Labs, an Australian-based startup dedicated to providing workshops, training, and developing technical products in AI security.

The foundational premise of Farlow's talk is rooted in the current landscape of AI adoption. She identifies an "inflection point" where AI technologies are no longer theoretical concepts but are being integrated into the core operations of diverse organizations, from financial institutions to critical infrastructure, and notably, casinos. This rapid deployment, however, is often characterized by a neglect of security considerations, leading to systems that are "inherently insecure." Farlow argues that this oversight persists largely because organizations tend to prioritize security only when confronted with substantial financial losses or severe damage to their brand reputation.

Casinos serve as an ideal, albeit provocative, analogy for this widespread problem. They are entities where brand trust and the perception of invincibility ("the casino always wins") are paramount to their business model. Consequently, any breach that undermines this trust or leads to significant financial payouts due to system manipulation would have catastrophic consequences. Farlow's collaboration with Casino Canberra further contextualizes this, providing a real-world backdrop to explore vulnerabilities in systems like facial recognition, which are critical for security and operational integrity within such environments. The talk, therefore, is not just about casinos but about illuminating a universal challenge in securing AI across all sectors.

Key Findings

▶ Watch: Harriet Farlow's 10-year journey in AI security and founding Mileva (2:00)

The central finding presented by Harriet Farlow is that despite the increasing reliance on Artificial Intelligence (AI) across various industries, including high-stakes environments like casinos, these systems are often deployed with significant and inherent security vulnerabilities. Her work explicitly aimed to test this hypothesis by focusing on the facial recognition AI systems commonly employed in casinos for surveillance, security, and customer identification. The primary discovery is that these systems, vital for maintaining security and operational integrity, are susceptible to various forms of attack, challenging the notion of their robustness and reliability.

Farlow’s research and practical demonstrations underscore that the perceived infallibility of AI, particularly in critical applications like security, is a dangerous misconception. She highlights that the rush to adopt AI technologies often leads to overlooking crucial security considerations in their design, development, and deployment lifecycle. This oversight creates fertile ground for adversaries to exploit weaknesses, not necessarily through traditional cyberattack vectors, but through methods specifically targeting the unique characteristics of machine learning models. The objective to "hack the facial recognition AI" directly translates into the finding that such systems are indeed vulnerable to manipulation, potentially allowing unauthorized access, evasion of surveillance, or even misidentification of individuals.

Furthermore, the talk implicitly reveals that many organizations, including casinos, lack a comprehensive understanding of the specific threats posed by AI exploitation. The focus tends to remain on traditional cybersecurity threats, while the unique attack surface introduced by AI models remains largely unaddressed. This gap in understanding and preparedness means that even sophisticated organizations with robust conventional cybersecurity defenses may be critically exposed to AI-centric attacks. Farlow's work serves as a critical validation of the urgent need for a dedicated focus on AI security, demonstrating that the potential for financial loss and reputational damage due to AI vulnerabilities is not theoretical but a present and tangible risk.

Technical Deep Dive

▶ Watch: Main objective: Hacking the casino's AI systems (3:08)

While the provided transcript primarily focuses on the conceptual framework and speaker's background, Harriet Farlow's objective to "hack the facial recognition AI" in a casino context strongly implies a deep dive into specific AI security attack vectors. Given her expertise in machine learning security, the technical content would undoubtedly explore methods to compromise the integrity and reliability of such systems. These methods typically fall under the umbrella of adversarial machine learning.

Facial recognition systems are essentially classification models that take an image of a face as input and output an identity or a confidence score regarding a match. Their vulnerability stems from the fundamental way neural networks learn patterns from data. Attackers can exploit this learning process or the model's inference phase.

One prominent attack vector is adversarial examples. These are inputs (images, in this case) that have been subtly perturbed with imperceptible noise, designed to fool the AI model into misclassifying them while remaining visually identical to a human observer. For facial recognition, an attacker could generate an adversarial image of their face that the system identifies as someone else (e.g., a VIP or a banned individual), or conversely, an image of a legitimate person that the system fails to recognize or misidentifies as the attacker. Common techniques for generating these include the Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), or Carlini and Wagner (C&W) attacks. These methods compute gradients of the model's loss function with respect to the input image, then adjust pixel values in the direction that maximizes misclassification. While the specific tools used by Farlow were not detailed in the provided transcript, open-source frameworks like Foolbox or CleverHans are commonly used for generating such examples.

Another significant threat is data poisoning. This attack occurs during the training phase of an AI model. If an adversary can inject malicious or manipulated data into the training dataset, they can subtly alter the model's learned patterns, leading to degraded performance or biased decision-making in specific scenarios. In a casino context, this could involve injecting images of individuals with manipulated labels into the facial recognition system's training data. For example, regularly associating an image of a known card counter with a "trusted customer" label could gradually poison the model, making it less effective at identifying that individual or similar individuals over time. This type of attack is particularly insidious because it compromises the model's integrity at its foundation, making detection challenging once the model is deployed.

Model evasion attacks are closely related to adversarial examples but focus on altering inputs at inference time to avoid detection. For facial recognition, this might involve an attacker subtly modifying their appearance (e.g., specific glasses, makeup, or even projected light patterns) to trick the system without being overtly obvious to human observers. Research has shown that even small, strategically placed stickers on a person's face can cause state-of-the-art facial recognition systems to misidentify individuals or fail to recognize them altogether. These physical adversarial examples highlight the real-world applicability of these theoretical attacks.

Model inversion attacks and membership inference attacks are also relevant, though perhaps less directly for "hacking" in the Ocean's 11 sense. Model inversion aims to reconstruct sensitive training data (e.g., a person's face) from a deployed model, potentially exposing private biometric information. Membership inference determines if a specific data point was part of the model's training set, which could reveal whether a particular individual is in the casino's database of known individuals. While the provided transcript doesn't specify these, they are crucial considerations for the overall security and privacy of facial recognition systems.

The lack of specific tool names, CVE numbers, version numbers, or percentages in the provided introductory transcript snippet means that the exact technical methodology employed by Farlow for her demonstration is not detailed here. However, based on the stated objective of "hacking the facial recognition AI" and the general landscape of AI security research, it is highly probable that her technical deep dive would have elaborated on these or similar adversarial machine learning techniques, showcasing how subtle manipulations can lead to significant security breaches in deployed AI systems. The core takeaway from a technical perspective is that AI models are not black boxes immune to manipulation; rather, their mathematical underpinnings and data dependencies present novel attack surfaces that require specialized defensive strategies.

Demo / Proof of Concept

▶ Watch: Talk plan: explore, hack facial recognition AI, lessons learned (6:10)

While the provided transcript is introductory and does not detail the specifics of the demonstration, Harriet Farlow explicitly stated her objective was to "hack the facial recognition AI" in a casino context. This strongly implies a Proof of Concept (PoC) or live demonstration that showcased the practical vulnerabilities of such systems. Given the nature of her work and the environment of DEF CON, it is highly probable that this involved a tangible illustration of how an attacker could bypass or mislead a facial recognition system.

A typical demonstration for "hacking facial recognition AI" would likely involve creating adversarial examples. This could manifest in several ways:

1. Evasion: An attacker presenting an image of themselves, or their actual face, that has been subtly altered (e.g., with specific patterns printed on a hat, glasses, or even projected light) such that the facial recognition system fails to identify them or misidentifies them as a generic "unknown" person. This would allow an individual, perhaps a known card counter or a banned patron, to enter restricted areas or operate unnoticed.
2. Impersonation: An attacker using an adversarial image or physical alteration to be misidentified as another individual. For instance, the system might be tricked into recognizing the attacker as a high-roller or a casino employee, potentially granting them access to privileges or areas they shouldn't have.
3. Denial of Service (DoS): Less likely for an "Ocean's 11" scenario but still a vulnerability, where an attacker could flood the system with inputs designed to overload its processing capabilities or cause it to crash, thereby disabling a critical security function.

The demonstration would likely have involved a simulated facial recognition system, perhaps using a publicly available model or a custom-trained model resembling those used in real-world casinos. The "hack" itself would have showcased the effectiveness of adversarial perturbations—small, often imperceptible changes to an image—in causing the AI model to make incorrect classifications. This could have been presented through live video feeds, static image comparisons, or even a hardware-based approach where a physical object (like a pair of glasses with a specific pattern) could fool the camera-based recognition.

The impact of such a demonstration would be profound: visibly illustrating that AI, despite its advanced capabilities, is not infallible. It would serve as a powerful validation of Farlow's core message that current AI deployments are "inherently insecure" and that organizations relying on these systems for critical security functions must proactively address these novel attack vectors. While the precise technical details of the PoC are not in the provided transcript, the stated objective ensures that the talk delivered a compelling, practical example of AI exploitation.

Defensive Implications

▶ Watch: Explaining why casinos are a perfect case study for AI security (6:30)

Harriet Farlow's talk carries significant defensive implications for any organization leveraging AI, particularly those in high-stakes environments like casinos. The core message is clear: the current approach to AI adoption often neglects security, leading to inherently vulnerable systems. Defenders must shift their mindset from traditional cybersecurity threats to actively address the unique attack surface presented by machine learning models.

Firstly, organizations must conduct AI-specific risk assessments. This goes beyond standard penetration testing and vulnerability scanning to evaluate the susceptibility of AI models to adversarial attacks, data poisoning, and other machine learning-specific threats. Understanding the potential impact of a compromised facial recognition system—whether it's misidentifying a high-value customer, failing to flag a banned individual, or granting unauthorized access—is crucial for prioritizing defenses.

Secondly, robust data governance and integrity measures are paramount. Since AI models are only as good as the data they're trained on, securing the data pipeline is critical. This includes strong access controls, encryption, and anomaly detection for training data, as well as rigorous validation processes to prevent the injection of malicious or biased data that could lead to data poisoning attacks. Regular auditing of training datasets and monitoring for data drift or unexpected changes can help mitigate this risk.

Thirdly, organizations should implement adversarial robustness techniques for their deployed AI models. This involves training models to be resilient against adversarial examples. Techniques include adversarial training, where models are trained on both clean and adversarial examples, and defensive distillation, which aims to smooth the model's decision boundaries. While no defense is foolproof, these methods can significantly increase the cost and complexity for attackers. Input sanitization and feature squeezing can also help detect and mitigate adversarial perturbations before they reach the model.

Fourthly, continuous monitoring and explainability are vital. Defenders need to monitor AI model performance not just for accuracy but also for unexpected behaviors or sudden drops in confidence that could indicate an ongoing attack. Implementing explainable AI (XAI) techniques can help security teams understand why a model made a particular decision, making it easier to identify and diagnose malicious manipulation or unintended biases.

Finally, there's a critical need for cross-disciplinary collaboration. Traditional cybersecurity teams must integrate with AI development and operations (MLOps) teams. Security professionals need training in AI fundamentals and attack vectors, while AI developers need to embed security-by-design principles from the outset. This holistic approach ensures that AI security is not an afterthought but an integral part of the AI lifecycle, preventing the deployment of "inherently insecure" systems and safeguarding critical operations from sophisticated AI-driven threats.

Key Takeaways

• AI Systems are Inherently Insecure: Many organizations are adopting AI without adequate security considerations, leading to systems that are vulnerable to exploitation from their inception.
• High-Stakes Environments are Prime Targets: Casinos, with their reliance on brand reputation and significant financial transactions, serve as a potent analogy for how AI vulnerabilities can lead to substantial monetary losses and reputational damage across any industry.
• Facial Recognition AI is Vulnerable: Systems like facial recognition, critical for security and operations, can be "hacked" using adversarial machine learning techniques, allowing for evasion, impersonation, or misidentification.
• New Attack Surfaces Require New Defenses: Traditional cybersecurity measures are insufficient; organizations must adopt AI-specific risk assessments, data governance, adversarial robustness techniques, and continuous monitoring.
• Proactive Security is Crucial: Waiting for a breach before prioritizing AI security is a costly mistake; embedding security-by-design and fostering collaboration between cyber and AI teams is essential for safe AI adoption.

About the Speaker(s)

Harriet Farlow is a prominent figure at the intersection of AI and security, with a decade of experience in the field. Originally from Australia, she holds an undergraduate degree in physics and transitioned into data science, working on defense projects before moving to the United States for a startup venture. Her deep commitment to the field led her to pursue a PhD in machine learning security, which she completed in 2021. Recognizing a significant gap in the industry's focus on AI security, Farlow founded Mileva Security Labs, an Australian-based startup dedicated to providing specialized workshops, training, and developing technical products in AI security. She is known for her practical approach to AI vulnerabilities, as demonstrated by her collaboration with Casino Canberra, and can be found sharing her insights on social media under the handle @HarrietHacks.