Against the Tyranny of Optimization: On the Stability of Automated Republics

Against the Tyranny of Optimization: On the Stability of Automated Republics

Speakers: Katie Moussouris

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=Z7WE8XPqO70

Overview

In her compelling keynote at BSides SF, Katie Moussouris, founder and CEO of Luta Security, delivered a stark warning about the societal implications of unchecked technological advancement, particularly in the realm of Artificial Intelligence. Titled "Against the Tyranny of Optimization: On the Stability of Automated Republics," her talk explored how the relentless pursuit of efficiency and growth, driven by AI and automation, is concentrating power, eroding worker rights, and challenging the very foundations of democratic societies. Moussouris masterfully connected historical precedents of industrial revolutions with the current AI era, highlighting the potential for widespread societal upheaval if proactive measures are not taken.

Moussouris argued that while philosophers historically warned against political tyrants, the modern threat emerges from "dashboards and automation" that represent an unprecedented accumulation and concentration of power. She posited that the speed at which AI-driven decisions are being made far outpaces the ability of traditional institutions, organizations, and governments to absorb and respond to them. The talk served as a crucial call to action for cybersecurity professionals, urging them to recognize that their role extends beyond technical defense to actively engaging in policy-making to safeguard human well-being and societal stability in an increasingly automated world.

The significance of this talk lies in its ability to bridge technical cybersecurity concerns with broader economic and political discourse. Moussouris demonstrated how trends observed within the bug bounty ecosystem—her self-professed "special interest"—serve as a canary in the coal mine for larger societal shifts. By illustrating how AI impacts vulnerability disclosure, labor markets, and wealth distribution, she underscored the urgent need for technologists to participate in shaping the future, advocating for shared prosperity over mere technological dominance.

Background

▶ Watch: Introduction to 'tyranny of optimization' and 'automated Republics' (0:00)

The problem Moussouris addressed is rooted in the accelerating pace of automation and Artificial Intelligence (AI) across all sectors, leading to an unprecedented concentration of power. Historically, philosophers warned against tyranny from authoritarian governments; today, this threat manifests through the seemingly benign interfaces of dashboards and automated systems that make decisions faster than human institutions can comprehend or regulate. In cybersecurity, this pattern is familiar: attackers adapt first, defenders play catch-up, and governments lag furthest behind. The advent of abundant intelligence, as exemplified by tools like Claude being used in CTFs, signals a new phase where this gap is rapidly widening.

Moussouris used the evolution of bug bounties and vulnerability disclosure programs as a prime example of this phenomenon. Initially conceived to democratize security, invite hackers, and uncover novel vulnerabilities, bug bounties faced systemic issues even before the widespread adoption of AI. Many organizations, particularly institutions, struggled to process the volume of reports, often finding that over half of reported vulnerabilities were known CVEs for which patches had not been applied. For researchers, the gig economy model of bug bounties created an unsustainable career path, rewarding speed and volume over consistent income, except for an elite minority.

The advent of AI exacerbated these pre-existing issues, introducing what Moussouris, referencing Corey Doctorow's "enshittification," termed the "en-slopification" of bug bounties. Automated tools and AI-generated reports, often hallucinated or low-quality, flooded platforms, overwhelming organizations. This problem is not confined to bug bounties; it reflects a broader societal trend where the gig economy has already eroded worker rights and bargaining power, as seen with Uber drivers. The current economic landscape, characterized by a K-shaped economy where those with capital accumulate wealth while others struggle, further concentrates power. Moussouris highlighted that 40% of the United States' GDP in the past year was reliant on just seven companies, many of which are fueled by AI growth, signifying a dangerous centralization of economic power. This backdrop sets the stage for a potential industrial revolution with profound and potentially destabilizing societal consequences.

Key Findings

▶ Watch: Abundance of AI intelligence and its use in CTFs (3:00)

Moussouris presented several critical findings regarding the impact of AI on cybersecurity and society:

• AI's Dual Impact on Bug Bounties: AI has created a deluge of "AI slop"—low-quality, often hallucinated vulnerability reports—that overwhelms human triage teams and platforms. Simultaneously, advanced AI systems are capable of discovering genuinely novel and critical vulnerabilities at scale, a capability previously thought to be years away. This dual effect strains organizations' capacity to manage vulnerabilities.
• Organizational Vulnerability Management Deficiencies: Many organizations lack an "internal digestive system of bugs," meaning they are ill-equipped to process, prioritize, and remediate the sheer volume of vulnerability reports, even for known CVEs. The influx from AI-driven discovery exacerbates this fundamental weakness.
• Gig Economy Dynamics and Automation: Bug bounty platforms, operating on a gig economy model, are inherently incentivized by market forces to favor automation and reduce reliance on human hackers, despite the stated intentions of their operators. This dynamic threatens the sustainability of bug bounty hunting as a human profession.
• Autonomous AI Vulnerability Discovery is Real: Moussouris cited concrete examples, such as Expo topping HackerOne leaderboards by autonomously finding and exploiting vulnerabilities (with human review for platform compliance), and Isle discovering over 16 OpenSSL CVEs. Even Google's AI has submitted valid vulnerabilities to projects like FFmpeg, demonstrating AI's advanced discovery capabilities, albeit sometimes without accompanying patches, creating "CVE slop" for maintainers.
• Concentration of Wealth and Power: The current AI revolution is accelerating a K-shaped economy, where wealth and capital are increasingly concentrated. Moussouris noted that seven mega-corporations account for 40% of the U.S. GDP, signifying a dangerous reliance on a few entities.
• Public Subsidies for AI Growth: The public is already subsidizing the AI revolution through both intellectual labor (training models through interaction) and direct financial costs. Moussouris highlighted that increased utility bills are a direct consequence of the massive power consumption required by new AI data centers, with an estimated $9.3 billion in costs passed onto consumers for infrastructure build-out. For example, Mid-Atlantic grid prices spiked 63% in 2025 due to these demands.
• Government Policy Prioritizes Dominance Over Safety: The US government's AI legislative framework, as observed by Moussouris, prioritizes "dominance" in AI development and discourages state-level regulation. This approach optimizes for growth and technological leadership at the potential expense of societal safety and stability, reflecting a short-sighted view of long-term consequences.

Technical Deep Dive

▶ Watch: Bug bounties: initial promise vs. pre-AI problems (4:00)

The technical core of Moussouris's argument revolves around the capabilities of modern AI in vulnerability discovery and exploitation, and the systemic challenges this presents. The talk illustrated how AI is not merely assisting humans but is becoming an autonomous agent in the cybersecurity landscape.

At the heart of AI's impact is its ability to perform advanced vulnerability research and exploit development. Systems like Expo, developed by leading hackers, demonstrate a full lifecycle capability: autonomously identifying potential vulnerability candidates, conducting experimentation to confirm weaknesses, writing functional exploits, validating their efficacy, and finally, generating detailed vulnerability reports. While Expo still requires human review for compliance with platforms like HackerOne, its underlying automation signifies a profound shift towards AI-driven security testing.

Another compelling example is Isle, co-founded by Jay Baloo, which has been credited with discovering over 16 novel OpenSSL CVEs. These vulnerabilities were not obscure but "hiding in plain sight" within open-source codebases, underscoring AI's capacity for deep, scalable analysis that often surpasses human capabilities in terms of speed and thoroughness. This class of technology, barely conceivable a year prior, now represents a significant force in uncovering critical flaws.

Even tech giants are leveraging AI for vulnerability discovery. Google, for instance, has utilized its AI to submit valid vulnerabilities to open-source projects like FFmpeg. However, Moussouris pointed out a critical flaw: Google's AI was not coupled with its CodeMender AI patch creator at the time. This resulted in "CVE slop" for volunteer-driven open-source projects, as they received a high volume of valid bug reports without corresponding patches, overwhelming their limited resources. This highlights that while AI excels at discovery, the ecosystem's capacity for remediation remains a bottleneck.

The influx of AI-generated content has had tangible effects on bug bounty program operations:

• The Curl project famously shut down its bug bounty program, citing "death by a thousand slops" due to the overwhelming volume of low-quality, automated reports. They maintained their vulnerability disclosure program but couldn't sustain the bounty model under AI pressure.
• The Node.js program implemented a minimum reputation bar for bug submissions, an attempt to filter out "slop" but potentially disenfranchising new, legitimate bug hunters.
• Bugcrowd updated its terms of service to warn and eventually ban users found to be abusing the system with AI-generated submissions.

Moussouris stressed that the fundamental constraint in vulnerability management has never been solely discovery but the "internal digestive system of bugs"—the organizational processes for triage, validation, remediation, and patching. Most organizations lack the robust internal mechanisms to handle the existing volume of vulnerabilities, let alone the exponential increase driven by AI. This systemic weakness is being exacerbated at scale, turning a technical triumph (AI discovery) into an institutional crisis.

Furthermore, the operation of these advanced AI systems demands immense infrastructure. AI data centers are massive consumers of energy, leading to significant increases in utility bills for the general public. Moussouris cited a 63% spike in Mid-Atlantic grid prices in 2025, with an estimated $9.3 billion in costs passed onto consumers to support this infrastructure build-out. This illustrates a hidden technical cost of AI development, where the public indirectly subsidizes the growth of powerful corporations.

Demo / Proof of Concept

▶ Watch: Unsustainable career path for most bug bounty researchers (6:00)

While the talk did not feature a traditional technical demonstration of AI finding vulnerabilities or exploiting systems, Katie Moussouris presented a unique, performative proof of concept to illustrate AI's current capabilities and limitations in creative tasks. She shared an AI-generated song, written in the style of the musical Hamilton, titled "You'll Adapt." Moussouris explained that she had to "coax AI to make that song rhyme" and required "significant amount of human intervention to produce the lyrics." She then sang the song herself, acknowledging that the AI could not perform it.

This "musical" interlude served to highlight that despite AI's advancements, human intervention remains crucial for refining outputs and overcoming current limitations, particularly in nuanced or creative domains. It demonstrated that while AI can generate content, the "human in the loop" is still a necessity, both for safety and for achieving desired quality, especially at this stage of AI's evolution. This artistic demonstration acted as a meta-commentary on the broader theme of the talk: AI's dependence on human intellectual labor and the ongoing, albeit shifting, role of humans in an automated world.

Defensive Implications

▶ Watch: AI's 'en-slopification' and overwhelming bug bounty platforms (7:00)

The implications of Moussouris's talk for defenders extend far beyond traditional technical controls, urging a holistic re-evaluation of strategy from the enterprise level to societal governance.

For Organizations and Security Teams:

The most immediate defensive implication is the urgent need to bolster internal vulnerability management processes. As Moussouris articulated, simply opening a bug bounty program or relying on AI for discovery without a robust "internal digestive system of bugs" is akin to opening a wide mouth without intestines. Defenders must invest in sophisticated triage, validation, and remediation workflows to handle the increased volume and potential "AI slop" from automated reporting. This includes:

• Automated Triage and Prioritization: Implementing AI-assisted tools for initial filtering of reports to distinguish genuine novel vulnerabilities from known CVEs or hallucinated submissions.
• Efficient Patch Management: Streamlining internal processes to apply patches for known vulnerabilities more rapidly, reducing the attack surface that AI-driven scanning tools will inevitably find.
• Resource Allocation: Re-evaluating security team structures and budgets to allocate resources not just for discovery, but for the full lifecycle of vulnerability resolution.

For Cybersecurity Professionals:

The nature of cybersecurity work is evolving. While expert "human-in-the-loop" roles will remain critical for the foreseeable future, especially for complex analysis, ethical oversight, and strategic decision-making, the talk suggests a potential erosion of entry-level and routine security tasks due to AI. Professionals must:

• Adapt and Upskill: Focus on developing skills that complement AI, such as critical thinking, complex problem-solving, ethical reasoning, and interdisciplinary understanding.
• Embrace Policy Engagement: Moussouris's most profound call to action is for cybersecurity professionals to become active participants in AI policy and legislation. She stressed that "your job was always more than just saving computers; it was about the people that depend on them." This means getting "in the room" where AI regulations are being drafted, providing expert, timely input, and advocating for policies that prioritize societal stability and shared prosperity over unchecked technological dominance. This includes challenging regulatory frameworks that solely optimize for AI growth without considering safety or equitable distribution of benefits.

Broader Societal Defense:

Beyond technical and professional responses, Moussouris highlighted the need for collective action to defend against the "tyranny of optimization" at a societal level:

• Advocacy for Worker Protections: Recognizing the erosion of worker rights in the gig economy, defenders should support policies that provide protections for workers in the evolving AI-driven labor market.
• Equitable Wealth Distribution: Engaging in discussions around mechanisms like Universal Basic Income (UBI) as a "dividend" for intellectual labor and societal contributions, rather than charity, to mitigate the economic dislocations caused by widespread automation.
• Challenging Unchecked Dominance: Actively questioning and advocating against government policies that prioritize "dominance" in AI development at the expense of safety, ethical considerations, and democratic stability. The cybersecurity community, with its deep understanding of systemic risks, is uniquely positioned to articulate these dangers.

Ultimately, the defensive implications underscore that cybersecurity in the age of AI is no longer a purely technical discipline. It requires a profound engagement with economic, social, and political structures to ensure that the "automated Republics" we are building remain stable and serve the well-being of all citizens, not just the interests of a concentrated few.

Key Takeaways

• AI Accelerates Societal Upheaval: The rapid advancement and unchecked optimization of AI are not just technological shifts but drivers of a new industrial revolution with the potential for unprecedented societal and economic instability, reminiscent of past industrial revolutions but at a greater scope, depth, and speed.
• Bug Bounties as a Warning Sign: The bug bounty ecosystem serves as a microcosm of AI's broader impact, demonstrating how AI generates both valuable novel vulnerability discoveries and overwhelming "AI slop," exacerbating organizations' existing struggles with vulnerability management.
• Concentration of Power and Wealth: The "tyranny of optimization" for AI growth is leading to a dangerous concentration of economic power (e.g., 40% of US GDP from 7 companies) and a K-shaped economy, where profits are privatized while costs (like utility bills for data centers) are socialized.
• Human Intervention Remains Critical (for now): Despite AI's rapid progress, human intellectual labor is still essential for training models, refining outputs, and providing crucial oversight, underscoring the current necessity of the "human in the loop" for both safety and quality.
• Technologists Must Engage in Policy: Cybersecurity professionals have a moral imperative to move beyond technical defense and actively participate in shaping AI policy and legislation. Their unique understanding of systemic risks is vital to advocating for shared prosperity, worker protections, and societal stability over mere technological dominance.
• A Crossroads for Humanity: We are at a critical juncture where AI can either be leveraged to create an engine of unprecedented shared prosperity or, if left unregulated and optimized solely for growth, become the fastest accelerator towards societal revolution and widespread human suffering. The time to act and organize is limited while human labor still holds leverage.

About the Speaker(s)

Katie Moussouris is the founder and CEO of Luta Security, a company dedicated to creating and managing sustainable bug bounty and vulnerability disclosure programs. She is a highly respected voice in the cybersecurity community, known for her pioneering work in vulnerability disclosure, bug bounties, and responsible security practices. Moussouris advises both companies and governments on emerging trends in Artificial Intelligence and cybersecurity, bringing a unique blend of technical expertise and policy insight to critical discussions about the future of technology and society. Her work consistently emphasizes the human element in cybersecurity and the broader societal implications of technological advancements.