Keynote: Perspectives on Trust in Hardware Supply Chains

Keynote: Perspectives on Trust in Hardware Supply Chains

Speakers: Bunnie Huang, Renowned Hardware Hacker and Designer

Conference: Black Hat Asia

YouTube: https://www.youtube.com/watch?v=Nv92TuocnwA

Overview

In this thought-provoking keynote at Black Hat Asia, renowned hardware hacker and designer Bunnie Huang delves into the intricate and often overlooked challenges of trust within global hardware supply chains. The talk dissects the economic incentives that shape the landscape of hardware attacks, revealing that the most prevalent threats are not always the sophisticated, nation-state-level implants often sensationalized in media, but rather pervasive, economically driven fraud schemes. Huang argues that while these "mundane" attacks may seem less glamorous, they are fostering a highly capable adversary with advanced hardware manipulation skills, posing a significant latent risk for future, more targeted information security breaches.

Huang's presentation challenges conventional wisdom, urging the security community to "follow the money" to understand the true nature of hardware supply chain vulnerabilities. He illustrates how the unique economics of hardware production and distribution create different incentives for malicious actors compared to software. By sharing personal anecdotes and deep insights gained from years of working within global manufacturing hubs like Shenzhen, Huang paints a vivid picture of a dynamic, incremental cat-and-mouse game between attackers and defenders, where the current state of defense is alarmingly underdeveloped, especially against highly sophisticated chip-level modifications.

The talk ultimately serves as a critical call to action, emphasizing that while current defenses are insufficient, solutions are possible through greater transparency, full-stack collaboration, and the adoption of novel inspection techniques. Huang's expertise, honed from breaking the original Xbox's hardware security to pioneering open hardware initiatives, provides a unique and authoritative perspective on a problem that is only growing in complexity and criticality as more of our digital lives become rooted in physical silicon.

Background

▶ Watch: Introduction to chaos, diversification, and distributed systems (0:00)

The increasing reliance on hardware as a root of trust for critical systems, from cloud infrastructure to personal devices, has brought the security of the hardware supply chain into sharp focus. As the conference host notes in the introduction, strategies for resilience in chaotic times often involve simplifying stacks and moving trust into hardware, such as using Hardware Security Modules (HSMs) for DNSSEC keys. However, this shift inherently raises a crucial question: how do we truly analyze and ensure the security of this underlying hardware?

Bunnie Huang begins by illustrating the immense complexity of the global hardware supply chain. Far from a simple linear process, it involves numerous stages—chip fabrication, packaging, distribution, gray markets, box build, customs, and retail—each presenting multiple points of vulnerability. This intricate, globalized ecosystem makes it challenging to maintain integrity. While high-profile incidents like the 2018 "Big Hack" story by Bloomberg have brought attention to potential hardware implants, Huang contends that the focus has often been misplaced, searching for "smoking guns" in the wrong places.

A core tenet of Huang's analysis is to always "follow the money" and examine the economic incentives driving threat actors. He contrasts the economic curves of software and hardware. Software profitability often starts negative, requiring significant investment to build volume and network effects before becoming highly profitable (e.g., ransomware, which can generate billions annually, impacting millions). Hardware, conversely, is most profitable at the very first unit sold; the highest price is commanded at launch, and profitability typically declines with competition. This fundamental difference in economic models dictates the types of attacks that are most attractive to a "numerical majority" of threat actors. Rather than investing in complex, high-risk exploit chains for servers or cloud roots, most actors are "too busy making money on much simpler attacks."

Key Findings

▶ Watch: Introduction of keynote speaker Bunny Hong (4:20)

Bunnie Huang's key findings revolve around the economic drivers of hardware supply chain attacks, revealing a pervasive and often underestimated threat landscape. He identifies warranty fraud as a prime example of a simple, yet highly profitable, hardware attack that currently dominates the landscape. This type of fraud exploits common manufacturing defects and the economic value of spare parts.

The process of warranty fraud, as detailed by Huang, typically involves four steps:

1. Discovering a common manufacturing defect: Repair technicians, often earning low wages (e.g., $3/hour in assembly lines), naturally identify recurring product failures as part of their job.
2. Figuring out how to trigger the error code: Information sharing within repair communities (e.g., chat groups) enables technicians to learn how to reliably induce specific error states.
3. Assembling devices from scrap parts to replicate the error: Repair persons accumulate cores and components from broken devices, effectively mining e-waste for valuable spare parts. This "tailings of the slaughter" are then reassembled into "Franken phones" designed to exhibit the target defect. Huang highlights the "original hackerism" ethos prevalent in Shenzhen, where nothing goes to waste, and every possible value is extracted from discarded electronics. He describes scenes of individuals extracting silicon chips from motherboards in open markets, turning e-waste into a "gold mine" with bins containing hundreds of thousands of dollars in parts.
4. Returning the defective device for a new one via warranty fraud: The assembled Franken phone is then used to claim a warranty replacement, often facilitated by "fencers" who handle the actual exchange, amplifying a few hundred dollars of scrap value into a $1,000 payday.

A specific example Huang cites is the iPhone 6 Error 53, caused by an authentication failure with the secure enclave. This legitimate field failure was easily inducible, allowing fraudsters to return bare-minimum phones (e.g., with low-grade batteries, steel ballast for weight, or screens with bad pixels hidden by a black display) to receive brand new devices. Apple reportedly lost billions of dollars to warranty fraud during this period, with single fraud rings still clearing up to $10 million today—a figure comparable to ransomware payouts, but targeted at a single company. This also explains why stolen phones from places like London often reappear powering up in Shenzhen, feeding a massive repair and recycling market that thrives on spare parts.

Huang also introduces the concept of a spectrum of authenticity for hardware. It's not simply "fake" or "real." This spectrum ranges from:

• Total Fakes: Made with fake processes and fake parts (e.g., a 5-cent chip in a $5 package). These are easily caught because they often don't work.
• Clones and Copies: Better quality fakes, but still distinct from genuine.
• Franken Phones: Made with real parts (e.g., genuine cases, some real board components) but assembled from salvaged or unauthorized sources.
• Relabeled Parts: Genuine silicon that has been relabeled to obscure its true nature (e.g., ES for engineering sample) or to misrepresent its specification. Huang shared a personal experience where 3% of FPGAs in a production run were relabeled engineering samples, a scheme that significantly increased distributor profits (from 5% to 8%, a 60% increase). This type of attack passes simple label checks but fails silicon ID verification.
• Ghost Shifting: This involves unauthorized production using legitimate factory equipment, materials, and processes, but outside of official production hours (e.g., workers stamping out USB connectors at 2 AM). The resulting parts are technically "real" in terms of manufacturing process but are unauthorized and often cut corners to maximize illicit profit.

These observations underscore that hardware threats are dynamic and local, with insertion rates that can be low (e.g., 3%) and highly targeted, making detection challenging. The adversary is diffuse, skills transfer across cells and geographical borders, and the informal nature of these organizations makes their true magnitude unknowable.

Technical Deep Dive

▶ Watch: Bunny Hong: Hardware supply chain complexity (6:10)

Bunnie Huang transitions from prevalent fraud to more sophisticated, chip-level attacks, categorizing them into four increasing levels of detection difficulty, using a compelling analogy of food adulteration.

Level 0: Relabeling Problems (Easy Detection, $0-$100 tools)

• Description: This involves obscuring a known defect or misrepresenting a part's quality. Huang's personal anecdote of receiving re-labeled engineering samples (ES) of FPGAs illustrates this. These were genuine Xilinx silicon but were not production-rated and meant to be scrapped. They were blended into the supply chain at a 3% insertion rate.
• Analogy: A vendor sticking a pricing label over a wormhole in a piece of fruit.
• Detection: Simple visual inspection, sometimes augmented by basic tests. However, the example of the relabeled FPGAs passing a "read the label" test highlights that even simple checks can be bypassed if they don't account for subtle modifications like blanks in engraving. True detection required reading the silicon ID via JTAG.
• Prevalence: Routinely practiced, but often overlooked in practice beyond basic label checks.

Level 1: Modified Network Interface Chip (Moderate Detection, $1,000-$10,000 tools)

• Description: This involves embedding a hardware Trojan into a chip, such as a PCI network interface chip (NIC), to exfiltrate data. Huang describes a hypothetical Trojan that, upon receiving an ICMP packet with a special key, would replay the last few network packets it observed.
• Analogy: Burying a foreign object like a razor blade or needle in fruit, detectable with a magnetometer.
• Implementation feasibility: Huang notes that all necessary blocks for such a Trojan (e.g., small CPU core, RAM, packet inspection logic) exist as free open-source IP blocks. Fabricating this on an older process node like 65 nanometers could be done for around $300,000, potentially even selling the modified chips for a profit.
• Detection: Visually detectable with a microraph. Huang shows a micrograph of a 65nm chip, indicating that a small CPU core (3.8mm wide) or even a few kilobytes of RAM would be "pretty obvious" on the chip's die.
• Prevalence: Practiced by targeted industries that know they are under attack and perform X-ray level checks.

Level 2: Modified CPU Pipeline (Hard Detection, $10,000-$100,000 tools)

• Description: This involves very subtle modifications within a CPU's pipeline to bypass security mechanisms, such as memory protection. Huang provides an example using a RISC-V CPU. The exploit targets the RA register (link register), which is primarily used for subroutine returns. If a load instruction uses RA as an address, the Trojan could strip out virtual memory translation and use a direct physical address fetch, effectively bypassing memory protection. This could be protected by a noop sequence as an unlock primitive to evade fuzzers.
• Analogy: Lacing an apple with strychnine – tasteless, odorless, requiring a chromatograph or advanced lab equipment to detect.
• Implementation feasibility: This exploit would require only "10 to 100 logic cells" within a large CPU block.
• Detection: Extremely difficult to detect visually. Huang explains that 100 gates might be a single "black dot" cluster on a micrograph, making it hard to discern from legitimate design elements, even with a good microscope.
• Prevalence: Academic papers exist on such threats, but practical detection methods are not widely deployed.

Level 3: Reduced Round Cryptography (Extremely Hard Detection, $100,000+ tools)

• Description: This is the most sophisticated and devastating attack, involving a minimal modification to cryptographic hardware to weaken its security without being detectable by standard means. Huang uses the example of an AES block cipher implemented in hardware, which typically cycles data through a single functional unit for multiple rounds (e.g., 14 rounds). The attack involves tying together the upper bits of holding registers in the round counter, effectively reducing a 14-round cipher to a 2-round cipher.
• Analogy: A genetically modified apple expressing an antibody that only attacks individuals with a specific antigen – detectable only by sequencing the apple's genome.
• Impact: This dramatically weakens the cryptography, making it vulnerable to differential cryptanalysis. Crucially, the timing side channel and power side channels remain similar, as the hardware still performs round functions in the dummy rounds.
• Detection: This is the "holy grail" of undetectable attacks. If a cryptographic enclave is built to gold standards (no scan chains, no inspectability, keys generated internally), the only way to test it is to encrypt and decrypt a block, which would still "pass" with reduced rounds. This type of attack can be implemented with just a single via edit within an incredibly complicated chip's intermediate metal layers. Huang states there is "no known mass deployable non-destructive method for detection" for such threats. Destructive analysis might identify a missing via, but this is impractical for mass inspection.
• Prevalence: Theoretical, with no known solutions for detection.

Huang concludes this section by noting that in practice, "no one is actually checking" for these threats beyond Level 0 label checks. Companies that do perform more rigorous checks often won't admit it due to competitive concerns, creating a veil of ignorance that allows threat actors "broad latitude to operate without any consequence."

Demo / Proof of Concept

▶ Watch: The reality of hardware supply chain attacks (7:20)

While the talk focuses primarily on attack vectors and the economic incentives driving them, Bunnie Huang does present a promising defensive technology as a proof-of-concept for future mitigation: infrared in-situ verification of silicon (Iris).

Huang describes Iris as a non-destructive method for inspecting certain types of chips after they have been attached to a circuit board. This technique allows for direct inspection of individual chips without requiring them to be removed or destroyed, moving beyond sample-based testing. He demonstrates this with a GIF showing a circuit board under regular lighting, then illuminated with infrared light. As the focus changes, the internal details of the chip become visible, shining through the silicon, which is transparent to infrared light. The grayscale image from this process reveals internal structures, offering a level of resolution suitable for detecting subtle modifications.

Crucially, Huang is committed to open-sourcing this technology, making it accessible to a wider audience. He claims that a DIY at-home setup for about $400 could enable users to detect Level 0 to Level 2 threats, and potentially Level 3 if the original design source is available for comparison. He provides a QR code linking to a "link tree" with details, papers, and resources for those interested in building their own Iris setup. This initiative aims to "raise the bar" for hardware inspection and empower end-users and smaller organizations to perform their own supply chain verification.

Defensive Implications

▶ Watch: First principle: Follow the money and incentives (8:20)

Bunnie Huang emphasizes that the current state of hardware supply chain defense is "generally undeveloped," but "defenses are possible." The core of effective defense lies in understanding the adversary's economic incentives and developing countermeasures that disrupt their profit models.

Here are the key defensive implications:

1. Acknowledge the True Threat Landscape: Defenders must move beyond the sensationalized "Big Hack" narratives and recognize that the vast majority of hardware supply chain attacks are driven by economic fraud, such as warranty fraud and the sale of re-labeled or ghost-shifted parts. These seemingly "mundane" attacks are building a pool of highly skilled hardware manipulation experts who could pivot to more sophisticated information security-related attacks if economic incentives shift.
2. Go Beyond Superficial Checks: Relying solely on label checks (Level 0 detection) is insufficient. As demonstrated by the relabeled FPGA example, even simple visual checks can be bypassed. Defenders need to implement more rigorous verification, such as reading silicon IDs and performing deeper analysis where possible.
3. Invest in Advanced Inspection Technologies: Technologies like infrared in-situ verification of silicon (Iris) offer a promising path forward. The ability to non-destructively inspect chips on a board significantly enhances detection capabilities for Level 0, 1, and potentially 2 threats. Widespread adoption and open-sourcing of such tools could democratize hardware security.
4. Embrace Transparency and Open Hardware: A significant barrier to defense is the lack of transparency in the supply chain and manufacturers' reluctance to admit problems. Huang argues for full-stack collaboration, starting with device manufacturers being willing to make devices inspectable. Open-source hardware is a crucial first step, providing reference designs for comparison and enabling greater scrutiny.
5. Develop Reference Standards and Inspectability: To effectively use inspection tools, there must be clear reference comparisons (what a genuine part should look like) and designs that facilitate inspection (e.g., specific packaging choices). This requires cooperation across the industry.
6. Address the Economic Tipping Point: Defenders must consider the factors that could shift the profit-to-risk ratio for threat actors, pushing them from warranty fraud to chip-level exploits. These factors include improved countermeasures (making fraud less profitable), macroeconomic concerns (e.g., a bust in the chip industry leading to empty fabs and lower entry barriers for chip design), or the involvement of Advanced Persistent Threats (APTs) or state-level actors who could "synergize with this existing ecosystem" by providing incentives or specific attack targets. A USB drive slipped into the hands of the right actor could be enough to trigger more sophisticated attacks.
7. No Magic Bullet: Huang stresses that there is no single solution like "hash and sign" for all hardware exploits. Effective defense requires a multi-faceted approach and continuous adaptation in an ongoing cat-and-mouse game.

Ultimately, the talk calls for a paradigm shift: from assuming hardware integrity to actively verifying it, from isolated efforts to collaborative defense, and from reactive measures to proactive design for inspectability and transparency.

Key Takeaways

• Economic Incentives Drive Hardware Attacks: Unlike software, hardware attacks are primarily driven by immediate profitability from individual units, leading to prevalent, "mundane" fraud rather than complex, large-scale exploits.
• Warranty Fraud is a Major Threat: Schemes like iPhone 6 Error 53 fraud have cost companies billions and demonstrate how easily manufacturing defects can be weaponized for profit, fostering a highly skilled hardware adversary.
• Hardware Authenticity is a Spectrum: There's no simple "fake vs. real"; threats range from relabeled genuine parts and "ghost shifting" to sophisticated chip-level modifications, each requiring different detection strategies.
• Chip-Level Attacks are Increasingly Sophisticated and Hard to Detect: From easily visible modified NICs (Level 1) to nearly undetectable reduced-round cryptography via a single via edit (Level 3), the difficulty and impact of chip modifications are escalating.
• Current Defenses are Insufficient: Most organizations only perform basic label checks, leaving them vulnerable to all but the most obvious fake parts. A culture of non-disclosure further obscures the true prevalence of hardware supply chain issues.
• Defenses are Possible but Undeveloped: Technologies like infrared in-situ verification (Iris) offer promising non-destructive inspection methods. However, widespread adoption requires full-stack collaboration, open-source hardware, and a willingness from manufacturers to design for inspectability.

About the Speaker(s)

Bunnie Huang is a renowned hardware hacker, designer, and author, recognized for his deep expertise in electronics manufacturing and supply chain security. He first gained prominence for his pioneering work in reverse engineering the original Microsoft Xbox, devising a way around its advanced hardware security system. This early work showcased his exceptional skill in uncovering and exploiting hardware vulnerabilities.

Huang later achieved "YouTube fame" by documenting his trips to chip fabrication plants and manufacturing facilities in Shenzhen, China. His efforts significantly demystified the complex world of Chinese manufacturing, opening doors for a new generation of creators and fabricators. He is highly active in the open RISC-V community and builds his own hardware products and custom projects for clients. His extensive experience across the entire hardware lifecycle, from design to fabrication and repair, provides him with a unique and authoritative perspective on the critical challenges of building and maintaining trust in global hardware supply chains.