Preventing One of The Largest Supply Chain Attacks in History

Preventing One of The Largest Supply Chain Attacks in History

Speakers: Maksim Shudrak, Security Researcher and Engineer, Bigte

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=Iv6VyOaG22c

Overview

Maksim Shudrak's DEF CON talk, "Preventing One of The Largest Supply Chain Attacks in History," unveils a critical and widespread supply chain vulnerability rooted in the recycling of cloud storage bucket names, specifically AWS S3. The presentation demonstrates how attackers can reclaim abandoned S3 buckets, inject malicious payloads, and subsequently compromise an enormous number of users, organizations, and even government networks globally. The core of the problem lies in software, scripts, or even malware that continue to reference S3 buckets long after their legitimate owners have deleted them, leaving a persistent, exploitable link.

Shudrak, a seasoned security researcher and red teamer, meticulously details the anatomy of such an attack, from initial discovery of vulnerable references to the sophisticated use of big data analysis and large language models (LLMs) to scale the exploitation. His research not only quantifies the alarming potential impact of this vector but also provides concrete examples of how it can lead to arbitrary code execution, data exfiltration, and the installation of backdoors across diverse environments. The talk serves as a stark warning about the overlooked risks in cloud infrastructure management and third-party dependencies, emphasizing the urgent need for both cloud providers and consumers to implement robust remediation strategies.

The significance of this research cannot be overstated. In an era where supply chain attacks are increasingly prevalent and devastating, Shudrak's work highlights a low-cost, high-impact attack vector that could lead to a global incident on the scale of major crypto worms or ransomware outbreaks. By demonstrating the practical exploitability across various software ecosystems—from PyTorch models to Maven packages and Chrome extensions—the talk compels the security industry to reconsider its approach to cloud resource lifecycle management and proactive vulnerability hunting. The findings underscore that even seemingly innocuous administrative actions, like deleting an S3 bucket, can inadvertently create a lasting security exposure if not managed with extreme diligence.

Background

▶ Watch: Imagining a massive crypto worm supply chain attack (0:20)

The motivation behind supply chain attacks stems from the increasing difficulty of penetrating network perimeters directly. As an attacker, the goal is to invert the cost-efficiency equation: achieve maximum impact with minimal effort. Supply chain attacks offer this advantage by targeting widely used components or services, allowing an attacker to reach privileged parts of networks or even production environments through a single point of compromise. The attack surface typically grows with the size and complexity of a company's software ecosystem, making these vectors particularly attractive for adversaries.

In the context of cloud buckets, the problem arises when a legitimate application or script running on a victim's machine is configured to pull executables, libraries, or other resources from a specific cloud bucket. If this bucket is later deleted by its owner, two critical conditions are met: the original content is no longer served (resulting in a 404 error), but crucially, the bucket name namespace becomes available for reclamation. An attacker can then claim this abandoned bucket name, upload malicious content, make it publicly accessible, and effectively hijack all subsequent requests from the original, legitimate references. This mechanism applies equally to malware, where a compromised host might pull a second-stage payload from an S3 bucket. If the original malware's C2 infrastructure is taken down and its buckets recycled, an unrelated attacker could claim those buckets and infect the original malware's victims.

This risk is not entirely new to the industry. Prior research and real-world incidents have highlighted similar vulnerabilities. For instance, backdooring Python packages, exploiting expired domains, and compromising popular npm packages or even top-level domains (TLDs) of sovereign states have all been demonstrated. Thomas Hansen's DEF CON talk two years prior specifically addressed the dangers of expired TLDs. Watchtower Labs also published research on hijacking numerous expired domains in web shells that continued to phone home to those domains. Most relevant to Shudrak's work is another Watchtower research effort that claimed 150 S3 buckets, showcasing the inherent risk. However, Shudrak's research significantly expands upon these prior efforts by investigating the true, large-scale impact, simulating the attack on a global level, navigating vast amounts of log data, and, critically, proposing root cause remediation strategies—areas that previous studies largely left unaddressed.

Key Findings

▶ Watch: Attacker motivation: why choose supply chain over perimeter? (2:50)

Maksim Shudrak's research uncovered an alarming scale of exploitable abandoned cloud buckets, demonstrating the potential for a truly massive supply chain attack. The sheer volume of data processed and the number of affected entities underscore the severity of this often-overlooked vulnerability.

His initial reconnaissance, spanning GitHub, Maven, PyPI, and malware analysis, yielded a substantial pool of potentially vulnerable targets:

• PyPI and Maven: Out of the packages crawled (approximately 40-45% of total), 480 abandoned S3 buckets were registered. Of these, 255 buckets received active traffic from 15,000 unique hosts across 129 countries, totaling 4.5 million requests and generating 3.5 GB of logs.
• Malware Infrastructure: By analyzing URLs found in malware samples, 188 buckets were registered. A staggering 175 of these received traffic from 20,000 hosts across 142 countries, resulting in 2 million requests and almost 4 GB of logs, predominantly for executables and JavaScript.
• GitHub: Searching GitHub repositories, 4,500 buckets were registered, with 1,500 receiving traffic. This source generated the largest volume, with 4 million requests, 19 GB of logs, and traffic from 243,000 hosts across virtually all countries (excluding North Korea).

Cumulatively, Shudrak's monitoring efforts collected 26 GB of logs and recorded 11 million requests. While many of these requests were for non-sensitive content like images or videos, sophisticated filtering and LLM-driven classification identified 500 sensitive buckets that posed a direct security risk.

The overall impact, derived from just five days of monitoring logs, was profound: an estimated 28,000 unique users across 158 countries were found to be potentially compromised. This included 134 distinct organizations and alarmingly, 25 government networks. The top affected countries were the US, Russia, China, and Germany, alongside other European nations.

A critical finding was the incredibly low cost of executing such a widespread attack. Shudrak estimated the total expense for log storage at approximately $20, suggesting that a determined attacker could launch this campaign with effectively a zero budget. This low barrier to entry makes the attack vector accessible to a wide range of threat actors.

The speaker also outlined various attacker profiles for whom this type of supply chain attack would be attractive:

• Activists: For disruption or to make a statement.
• Cybercrime Groups: For financial gain through ransomware, data exfiltration, or other illicit activities.
• Advanced Persistent Threat (APT) Groups: For sabotage, espionage, or long-term compromise of critical infrastructure.

Finally, the research revealed a "paradox of responsible disclosure" in this context: to identify the vulnerability, one must claim the bucket, thereby "fixing" it for the original reference. To avoid widespread harm, Shudrak collaborated with AWS on a remediation campaign, resulting in the successful securing of all identified sensitive buckets. However, he stressed that this is likely just the tip of the iceberg, with many more vulnerabilities potentially existing in other package repositories, GitHub commit histories, proprietary codebases, and across other cloud providers beyond AWS.

Technical Deep Dive

▶ Watch: Technical explanation of S3 bucket hijacking mechanism (3:30)

The technical foundation of this supply chain attack hinges on the recycling of cloud bucket names, particularly in AWS S3. When an S3 bucket is deleted, its name becomes available for re-registration by any other AWS user. If legitimate software, scripts, or even malware were previously configured to fetch resources from that specific bucket name, they will continue to make requests to it. An attacker can then register the abandoned bucket name, upload malicious payloads, and serve them to unsuspecting victims, effectively hijacking a previously established trust relationship.

Shudrak's methodology for discovering these vulnerable references was multifaceted:

1. GitHub: Leveraging Sourcegraph, a powerful code search engine, he queried for common S3 patterns like s3.amazonaws.com or s3 across millions of public GitHub repositories. Regular expressions were then used to extract specific bucket names from the search results, a process that took mere seconds.
2. Maven and PyPI: For package managers, the process was more intensive. It involved listing and fetching repositories from platforms like Repo1 and PyPI, downloading packages, extracting their contents in memory, and then applying regex to identify S3 bucket references. This phase alone took approximately 1.5 months of continuous processing, covering an estimated 40-45% of the total packages.
3. Malware: Identifying S3 references in malware was streamlined by using URLhaus, a resource that aggregates URLs found in malware samples. This allowed Shudrak to efficiently find links pointing to s3.amazonaws.com used by various malicious families.

Once potential bucket names were identified, Shudrak followed a strict rules of engagement: buckets were claimed but always kept private (returning a 403 error) to prevent actual compromise. For testing, clones of the buckets were used. Logs were enabled for five days to capture traffic, ensuring user privacy by not attributing to individual IPs or organizations.

A significant challenge was triaging the vast amount of log data—26 GB and 11 million requests. To filter out noise (e.g., requests for non-sensitive files like images, videos, or training data, and traffic from bots/scanners), and to aggregate requests from unique IPs, Shudrak employed a custom LLM agent. This agent, comprising a planner, aggregator, and classifier, was equipped with tools like internet search and GitHub search. It could analyze log entries, construct a context around each request (bucket name, IP, request type, object URI, referrer, user agent), and classify its sensitivity (e.g., executable, ML model, JavaScript). This LLM-driven approach reduced analysis time from potentially months to just a few days, identifying 500 truly sensitive buckets.

The research highlighted several distinct and highly impactful attack vectors:

• Executable Hijacking:
• Network Equipment Company: A large network equipment vendor previously served installation binaries from an S3 bucket. After changing their distribution policy and deleting the bucket, the old link remained active in client scripts (e.g., PowerShell). An attacker reclaiming this bucket could serve malicious executables, potentially compromising 162 hosts.
• Minecraft-Pie: A version of Minecraft for Raspberry Pi referenced an abandoned S3 bucket for package downloads. Hijacking this allowed for straightforward code execution on numerous devices.
• Vagrant VMs: Requests for Vagrant .box VM files from abandoned S3 buckets were common. Beyond backdooring the VM itself, Vagrant's configuration allows for system commands to be executed on the host machine when a box is added, providing an immediate path to host-level code execution.

• Poisoned PyTorch Models: A quarter of all sensitive requests targeted PyTorch models. PyTorch, by design, relies on pickle deserialization to load models. While some controls exist, pickle deserialization is inherently vulnerable to arbitrary code execution if untrusted data is processed. Shudrak demonstrated this with popular open-source projects like Super-gradients (4,000 stars) and another project with 33,000 stars. An attacker could inject malicious Python code into a PyTorch model file (.pth), and upon loading, the code would execute on the victim's machine.

• Maven and PyPI Package Backdooring:
• not_the_plea: This package, acting as a wrapper for a native binary, previously pulled its pre-built binary from an S3 bucket. By controlling this bucket, Shudrak could inject a backdoored library. Loading this library would lead to code execution, impacting five Fortune 500 companies, including three of the world's largest banks, an AV vendor, a global telecom, and a major retail company.
• Kuvadev: This Maven package referenced an additional host repository, which in turn pulled thousands of packages from an abandoned S3 bucket. This vector could lead to the installation of backdoor packages on five government entities in Europe and South America, a global investment fund, and two major tech/telecom companies.

• Malware Infrastructure Hijacking:
• Linker (Chrome Extension): This malicious Chrome extension, masquerading as a legitimate B9 application (e.g., a Flash playlist), pulled JavaScript from an S3 bucket to inject ads into victims' browsers. By reclaiming these S3 buckets, Shudrak demonstrated the ability to inject arbitrary JavaScript on websites visited by infected users (e.g., Gmail, Twitter, Facebook, LinkedIn). He highlighted that even with Content Security Policy (CSP) restrictions, data exfiltration was possible from sites like TikTok because s3.amazonaws.com is often allowed in CSP rules. This could impact thousands of users and government networks in 30 countries.
• RK Stealer: This data-stealing malware, active four years prior, pulled executables from S3 buckets over HTTP. Shudrak demonstrated that by reclaiming these buckets and using tools like FakeNet-NG, an attacker could serve any executable (e.g., a calculator) to compromised machines, which would blindly execute it.

Other sensitive request types included PHP and JavaScript files, static sensitive files (PDFs, Microsoft Office documents) serving as social engineering vectors (e.g., asking users to enable macros), and even data write attempts where misconfigured systems tried to log data into the reclaimed S3 buckets.

Demo / Proof of Concept

▶ Watch: Strict rules for ethical research and responsible disclosure (6:05)

Maksim Shudrak presented several compelling live demonstrations to illustrate the practical exploitability of abandoned S3 buckets.

1. PyTorch Model Poisoning:

• The demo began by showing an open-source project, Super-gradients (with 4,000 GitHub stars), attempting to load a pre-built PyTorch model from an S3 bucket. As the original bucket was abandoned, the script received a 403 Forbidden error, confirming the bucket's non-existence or private status.
• To simulate a successful attack safely, Shudrak applied a patch to the local Python script, redirecting the model loading attempt to a cloned S3 bucket under his control.
• He then introduced a small Python code snippet, adapted from research by Snook, designed to inject arbitrary code into a PyTorch model file. This snippet, when executed, would cause the system's calculator application to launch.
• The poisoned model (.pth file) was then uploaded to the reclaimed S3 bucket and configured for public access.
• Finally, rerunning the patched super-gradients script successfully loaded the poisoned model from the controlled S3 bucket, which in turn executed the injected code, causing the system calculator to pop up on screen. This clearly demonstrated arbitrary code execution through model deserialization.

1. Maven Package Backdooring (not_the_plea):

• Shudrak showcased an old version of the not_the_plea Maven package, which functioned as a wrapper for a native binary. When executed, this package attempted to pull a pre-built binary from an abandoned S3 bucket, again resulting in a 403 Forbidden error.
• Similar to the PyTorch demo, the package's internal reference to the S3 bucket was modified to point to a controlled, cloned bucket.
• A pre-compiled backdoored library, designed to trigger code execution, had already been uploaded to the controlled S3 bucket.
• Executing the modified not_the_plea package caused it to fetch and install the malicious library from the reclaimed bucket. Subsequently, running the associated binary (which would normally be installed by the package) led to arbitrary code execution on the machine. The speaker noted the significant impact this specific exploit could have on Fortune 500 companies and major banks.

1. Malicious Chrome Extension (Linker) JavaScript Injection:

• This demo focused on the Linker malware family, a Chrome extension that previously injected JavaScript from an S3 bucket into websites visited by victims.
• A malicious extension was shown installed in a browser.
• Shudrak navigated to several popular websites, including Gmail, Twitter, Instagram, Facebook, and LinkedIn.
• In the background, logs confirmed that the browser extension was attempting to pull JavaScript from the controlled S3 bucket and execute it within the context of these visited websites.
• To further illustrate the danger, a scenario was shown involving TikTok. Since s3.amazonaws.com is often permitted by Content Security Policies (CSP) on many websites, the injected JavaScript could exfiltrate sensitive data (e.g., cookies, page data) directly to the attacker's controlled S3 bucket, demonstrating a potent Cross-Site Scripting (XSS) vector.

1. RK Stealer Malware Hijack:

• The final demo replicated the hijacking of RK Stealer malware infrastructure within a Windows virtual machine environment, equipped with Wireshark and FakeNet-NG (a network simulator).
• A Wireshark installer, in this case, was a disguised malware sample. Running it, Wireshark showed network traffic as the malware attempted to pull a second-stage executable from an S3 bucket via HTTP.
• By configuring FakeNet-NG to respond to the malware's S3 requests, Shudrak intercepted the traffic. Instead of the original malware payload, FakeNet-NG was configured to serve a simple calculator executable.
• The malware, blindly executing whatever it received, launched the calculator, unequivocally demonstrating how an attacker could leverage abandoned malware infrastructure to deliver their own payloads for arbitrary code execution.

These demonstrations powerfully illustrated the diverse range of attack surfaces and the ease with which abandoned S3 buckets could be weaponized to achieve significant compromise, from direct code execution to widespread data exfiltration.

Defensive Implications

▶ Watch: Leveraging Sourcegraph to find S3 buckets in code (7:00)

The findings from Maksim Shudrak's research necessitate a multi-layered defensive strategy involving both cloud consumers and providers to mitigate the severe risks posed by abandoned cloud buckets.

For organizations and cloud consumers, immediate and proactive measures are crucial:

• Robust Static Source Code Analysis: Organizations must implement comprehensive static application security testing (SAST) tools to continuously scan their entire codebase, including internal applications and third-party dependencies, for hardcoded references to S3 buckets or other cloud resources. The focus should be on identifying URLs that point to s3.amazonaws.com or similar patterns, especially those that might become dangling.
• Dependency Monitoring and Management: Beyond internal code, a critical area of focus is on third-party libraries and packages. Developers should be educated on the risks of dependencies that pull resources from external, potentially unverified, cloud locations. Software Composition Analysis (SCA) tools can help identify such dependencies.
• Prompt Remediation of Dangling References: Upon identifying references to non-existent or abandoned buckets, organizations must prioritize removing these references or updating them to point to new, secure, and actively managed resources. This includes updating build scripts, configuration files, and application code.
• Utilize Existing Tooling: Shudrak noted that several decent tools already exist for detecting dangling S3 bucket references. Organizations should leverage these tools as part of their regular security hygiene and CI/CD pipelines.
• Secure Development Lifecycle (SDL): Integrate awareness of this vulnerability into the SDL. Developers should understand the implications of deleting cloud resources and the need to update or remove all associated references to prevent future exploitation.
• Network Egress Filtering: While not a complete solution, strict egress filtering can help prevent compromised systems from connecting to unknown or suspicious S3 buckets, potentially limiting the impact of an attack.

For cloud providers, particularly AWS, Shudrak proposed fundamental changes to enhance security by default:

• Secure by Default Cloud Design: The core issue stems from the recyclability of bucket names. Cloud providers should explore architectural changes that make such attacks inherently more difficult.
• Project-Scoped Bucket Names: A potential solution is to enforce project-scoped bucket names. Under this model, a bucket name would be tied to an immutable project identifier (e.g., project-name-bucket-name). If a project is deleted, its associated bucket names would also be permanently retired or locked, preventing another user from claiming them.
• Non-Recyclable Bucket Names: The most secure, albeit potentially challenging, long-term solution would be to make bucket names non-recyclable entirely. Once a bucket name is used, even if deleted, it should never be available for re-registration. Shudrak acknowledged potential caveats, such as the initial "gold rush" for desirable names or potential for abuse, requiring further research into critical user journeys.
• Higher Bucket Caps: AWS recently increased the maximum number of buckets per account to 10,000. While seemingly a minor change, Shudrak suggested this might indirectly reduce the problem by disincentivizing users from deleting buckets in the first place, thus reducing the pool of recyclable names.
• Proactive Management of Malware Buckets: For buckets identified as part of malware infrastructure, cloud providers should adopt a strategy similar to the Shadow Server initiative for malicious domains. These buckets should be permanently kept private or sinkholed, preventing any attacker from ever reclaiming them and reinfecting victims.
• Collaboration with Researchers: AWS's positive collaboration with Shudrak on remediation highlights the importance of cloud providers actively engaging with security researchers to identify and address systemic vulnerabilities.

In essence, while developers and organizations bear responsibility for managing their code and dependencies, the underlying architecture of cloud resource naming conventions presents a systemic risk. A collaborative effort between vigilant users and proactive cloud providers in adopting more secure-by-default paradigms is essential to prevent future large-scale supply chain attacks of this nature.

Key Takeaways

• Systemic Vulnerability: Abandoned cloud buckets, particularly AWS S3, create a critical supply chain vulnerability where attackers can reclaim deleted bucket names and serve malicious content to legitimate software, scripts, or malware still referencing them.
• Massive Scale, Low Cost: The research revealed a potential for widespread compromise, affecting 28,000 users, 134 organizations, and 25 government networks across 158 countries within just five days of monitoring, all at an estimated attack cost of merely $20.
• LLM-Augmented Exploitation: Advanced techniques, including big data analysis and custom LLM agents, can significantly automate and scale the discovery, classification, and weaponization of these vulnerabilities, reducing attacker effort from months to days.
• Diverse Attack Vectors: Exploitable scenarios range from arbitrary code execution via poisoned PyTorch models (leveraging pickle deserialization) and backdoored Maven/PyPI packages, to widespread JavaScript injection through compromised browser extensions (like Linker) and hijacking of old malware C2 infrastructure (like RK Stealer).
• Urgent Need for Remediation: Organizations must proactively scan their codebases and third-party dependencies for references to dangling cloud resources, promptly removing or updating them. Utilizing existing static analysis tools is crucial for this ongoing security hygiene.
• Cloud Provider Responsibility: Cloud providers should consider fundamental changes to bucket naming conventions, such as implementing project-scoped or entirely non-recyclable bucket names, and proactively managing buckets previously used by malware to enhance security by default.

About the Speaker(s)

Maksim Shudrak is a dedicated Security Researcher and Engineer currently working at Bigte. With a strong background in offensive security, he primarily focuses on red teaming engagements. Maksim holds a PhD in computer security, demonstrating his deep academic and practical expertise in the field. He is a returning speaker to the DEF CON stage, having presented twice before on various security topics. His commitment to the security community is further evidenced by his open-sourcing of several projects, including tools for phishing, software analysis, and GCP scanning.