Smart Devices, Dumb Resets:Testing Firmware Persistence in Commercial IoT

Smart Devices, Dumb Resets: Testing Firmware Persistence in Commercial IoT

Speakers: Matei Jose, Senior Penetration Tester, Happening XYZ

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=rLnlLLKISyY

Overview

In "Smart Devices, Dumb Resets," Matei Jose, a Senior Penetration Tester at Happening XYZ, delves into the critical security vulnerability posed by the inadequate sanitization of returned Internet of Things (IoT) devices by retailers. The talk highlights how readily available pre-owned or "repackaged" smart devices can harbor persistent, malicious firmware, even after users attempt "factory resets" or retailers conduct cursory checks. Jose's research demonstrates a clear pathway for attackers to backdoor consumer IoT devices, return them, and then have them resold to unsuspecting customers, creating a stealthy and scalable supply chain attack vector.

This research is particularly significant because it exposes a systemic flaw in the lifecycle management of IoT products, from manufacturing to retail and resale. As the proliferation of smart devices continues, the potential for backdoored devices to infiltrate homes and even corporate environments grows, posing risks ranging from privacy invasion (e.g., camera feeds from vacuum robots) to the establishment of botnets for widespread attacks. Jose's work serves as a crucial wake-up call for both retailers to enhance their security protocols for returned goods and for consumers to exercise extreme caution when purchasing pre-owned IoT hardware.

Background

▶ Watch: Introduction: Smart Devices, Dumb Resets (0:00)

Matei Jose's fascination with the tangible impact of virtual actions in the physical world, particularly within IoT, was sparked by earlier experiences. He recounts an incident at a conference where he easily escaped a self-service check-in kiosk mode, discovered it was running Windows, found the Wi-Fi password, and subsequently exploited badge printers using default "admin/admin" credentials. This multi-layered failure, involving people, processes, and technology, yielded a custom-printed badge—a physical souvenir from a digital exploit—solidifying his interest in IoT's real-world consequences.

The genesis of this specific research, however, dates back seven years to his student days when he observed how a returned, used inflatable kayak was quickly reposted and resold as "pre-owned." This experience, coupled with more recent inspirations from the IoT hacking community—including Andrew Bellini's beginner-friendly IoT talks, Michael Copola's work on backdooring router firmware (and his Router Post Exploitation Framework, RPF), Matt Brown's YouTube content, and Dennis Giza's DEF CON presentations on vacuum robot hacking—led Jose to formulate a critical hypothesis: "Retailers do not implement enough checks to prevent reselling backdoor devices."

To validate this, Jose devised a straightforward, five-step methodology:

1. Identify: Pinpoint devices with easily overwritable firmware.
2. Order: Purchase these devices.
3. Modify: Alter their firmware to include a persistent backdoor.
4. Return: Send the modified devices back to the retailer.
5. Re-acquire & Verify: Purchase the devices again when resold as pre-owned to confirm the firmware modification's persistence. The final step was crucial to prevent the spread of his proof-of-concept (PoC) backdoor into the wild, which he mitigated by using a web scraper to notify him of resold devices for immediate re-purchase.

Jose also considered the ethical implications, noting that while returning an item within policy isn't fraudulent, attempting to game the system for price drops can be unethical or illegal. His goal was not to exploit return policies for financial gain but to expose a security vulnerability within the retail supply chain.

Key Findings

▶ Watch: Talk origin: Kayak return and resale experience (4:00)

Matei Jose's research unequivocally confirmed his hypothesis: retailers, at least the five reputable ones he tested, indeed do not implement enough checks to prevent reselling backdoored devices. His findings revealed a concerning lack of security diligence in the return and resale process for IoT hardware.

Out of the 15 devices purchased from five different retailers (three from each), all 15 were successfully backdoored, returned, and subsequently resold as "repackaged" or "pre-owned." Jose was able to re-acquire 13 of these 15 devices thanks to his custom web scraper, which provided Discord notifications when his specific devices became available. Crucially, all 15 devices, including the two he couldn't re-purchase, eventually "called back" to his command-and-control (C2) server, proving that the malicious firmware persisted through the return process and into the hands of new, unsuspecting owners.

A significant finding was the complete absence of effective security checks by retailers. Before being resold, none of the devices initiated a C2 callback, indicating that the retailers' technical teams either did not connect the devices to the internet during their inspection or performed only rudimentary offline tests (e.g., checking if indicator lights blinked). This meant that even if the devices were checked for basic functionality, they would have appeared to work perfectly, allowing the backdoored firmware to slip through undetected. Furthermore, Jose noted that even devices he physically opened to pull firmware via SPI were accepted for return without comment, suggesting a profound lack of physical tampering detection.

The persistence of the custom firmware was a central discovery. Despite common assumptions that a "factory reset" button would restore a device to its original state, Jose's experiments showed that holding the reset button only removed configurations, leaving his custom backdoor firmware intact. This highlights a fundamental misunderstanding or misrepresentation of what a "factory reset" truly accomplishes on many consumer IoT devices. The research also validated his self-destruct mechanism, as the two devices he couldn't retrieve eventually stopped calling back after approximately 21 days, demonstrating that the backdoor could be designed to self-remove, mitigating long-term risk.

Technical Deep Dive

▶ Watch: Research hypothesis and methodology explained (6:40)

The technical execution of Matei Jose's research hinged on selecting easily modifiable IoT devices and employing specific techniques for firmware acquisition, modification, and re-flashing.

Device Selection and Firmware Acquisition:

Jose focused on OpenWRT compatible routers, specifically TPLink routers, for several strategic reasons. These devices are cheap, making them suitable for destructive testing without breaking the bank. More importantly, the OpenWRT community has extensively documented the process of installing custom firmware on them, confirming their susceptibility to firmware overwriting. He acquired 15 such devices from five different reputable retailers.

For firmware acquisition, Jose initially pursued two methods:

1. SPI (Serial Peripheral Interface): Physically opening a device and using an SPI programmer to pull the existing firmware directly from the memory chip. This method, while effective, was challenging due to the need for a steady hand and risked breaking physical tamper seals (though retailers surprisingly ignored this).
2. Publicly Available Firmware: For TPLink devices, firmware images are often available directly from the vendor's website, simplifying the acquisition process.

Firmware Analysis and Modification:

Once the firmware binary was obtained, Binwalk was used to analyze its structure, specifically to identify the embedded file system, which in this case was a squashfs file system. Jose extracted this file system to introduce his modifications.

The backdoor itself was designed to be minimal and stealthy for proof-of-concept purposes. Instead of a full agent, it consisted of a simple service configured to run every 10 minutes. This service would execute a GET request to a specific IP address belonging to Jose's EC2 instance, serving as his C2 server. Each request included a unique identifier, allowing him to track individual backdoored devices.

A critical aspect of the backdoor was its self-destruct mechanism. After 3,000 requests (approximately 21 days of continuous operation), the service was programmed to disable itself and delete the script, effectively restoring the device to its initial state. This "self-restore" functionality was vital for mitigating the risk of his PoC spreading indefinitely, particularly for devices he couldn't re-acquire. Jose also experimented with cosmetic changes, such as modifying the TPLink logo to his own, as a preliminary test of firmware modification capabilities.

Re-creating and Flashing Firmware:

After modifications, the firmware binary needed to be re-created. This was done using make squashfs. A crucial technical detail Jose encountered was the importance of specifying the correct compression method. The original TPLink firmware used XZ compression, while his initial make squashfs attempts defaulted to Zlib compression. This discrepancy could cause issues, highlighting the need to use the --compression flag to match the original firmware's compression method, ensuring the modified firmware was as close as possible to the original.

Flashing the modified firmware back onto the devices proved to be the most challenging step:

1. SPI: While capable, it was physically demanding and not practical for scaling or remote attacks.
2. Web Interface: Many devices offer a web interface for firmware updates. However, these often perform signature checks, preventing the upload of unsigned or modified firmware. Jose's attempts to upload via the web interface were unsuccessful due to these checks.
3. TFTP (Trivial File Transfer Protocol): This was the successful workaround. Many IoT devices, especially routers, include a TFTP client for recovery purposes. During boot-up or specific recovery modes, they might query a TFTP server on a predefined local IP address (e.g., 192.168.1.100) for a specific firmware file. By setting up a TFTP server with the modified firmware file at the expected location, Jose was able to trick the devices into pulling and flashing his custom image. This method was particularly effective because it bypassed the web interface's signature checks and did not require precise physical manipulation like SPI.

A key observation was that even after successfully flashing the backdoored firmware, subsequent attempts to "factory reset" the devices by pressing and holding the reset button only cleared user configurations and network settings, but the custom firmware remained persistent. This confirmed the deep-seated nature of the modification and the inadequacy of standard user-level reset functions.

Demo / Proof of Concept

▶ Watch: Simple proof of concept backdoor mechanism (8:00)

The entire research project served as a comprehensive proof of concept for the firmware persistence attack. Matei Jose meticulously executed his methodology, demonstrating the vulnerability from device acquisition to re-acquisition and C2 callback confirmation.

The demonstration unfolded as follows:

1. Device Acquisition: Jose purchased 15 TPLink routers from five different reputable retailers, acquiring three devices from each.
2. Firmware Modification: He physically (via SPI) and digitally (via TFTP) flashed his custom firmware onto these devices. The custom firmware contained a simple script that would send a GET request every 10 minutes to his EC2 instance acting as a C2 server. This script also included a self-destruct/self-restore mechanism designed to disable itself and delete the script after 3,000 requests (approximately 21 days).
3. Return Process: Jose initiated returns for all 15 devices, citing reasons like "changed my mind" or "looking for different devices." He noted that courier services picked up the hardware, and he received refunds within a week, with no additional questions asked, even for the device he had physically opened.
4. Monitoring and Re-acquisition: Jose deployed a custom web scraper that monitored the retailers' websites for his specific devices to be relisted as "pre-owned" or "repackaged." Upon receiving notifications via Discord, he attempted to re-purchase them immediately. He successfully bought back 13 out of the 15 original devices.
5. C2 Callbacks: Crucially, Jose received C2 callbacks from all 15 devices, including the two he was unable to re-acquire. This confirmed that the backdoored firmware persisted through the retailers' inspection process and that the devices were indeed resold to new customers. The callbacks from the two un-retrieved devices ceased after approximately one month, validating the effectiveness of his self-destruct script.

Jose highlighted visual evidence from the returned devices, showing "resealed tape on the outer edges of the box" and labels like "personalized security" on the packaging, ironically indicating a superficial re-packaging process rather than a thorough security check. The success of this PoC provided concrete evidence that retailers' existing return handling procedures are insufficient to prevent the reintroduction of backdoored IoT devices into the consumer market.

Defensive Implications

▶ Watch: Identifying suitable devices: OpenWRT compatible routers (8:30)

Matei Jose's research carries significant defensive implications for various stakeholders, from retailers and manufacturers to individual consumers and the broader security community.

For Retailers and Manufacturers:

The most immediate takeaway is the urgent need for retailers to overhaul their processes for handling returned IoT devices. Simply plugging in a device to check for blinking lights or basic functionality is grossly inadequate. Retailers should:

• Implement Rigorous Firmware Sanitization: For every returned IoT device, the firmware must be completely wiped and re-flashed with a cryptographically signed, known-good factory image. This process should be non-bypassable and verified.
• Conduct Comprehensive Functional and Security Checks: Devices should be connected to a segregated, isolated network during testing, never the retailer's internal network, to prevent any potential backdoors from gaining internal access.
• Inspect for Physical Tampering: Train staff to look for signs of physical manipulation, such as broken tamper seals, non-original screws, or unusual modifications to the casing. Jose's experience with an opened device being accepted for return highlights a major gap here.
• Educate Consumers: Clearly communicate what a "factory reset" truly entails for their devices, emphasizing that it often doesn't remove custom firmware.

Manufacturers, in turn, must design devices with security in mind:

• Secure Boot and Firmware Signing: Implement secure boot mechanisms and require all firmware updates to be cryptographically signed by the manufacturer. This would prevent unauthorized firmware from loading.
• Bootloader Locking: Lock down the bootloader to prevent unauthorized firmware flashing, especially via methods like TFTP or SPI, unless specific developer modes are intentionally enabled.
• Firmware Encryption: Encrypt firmware images to make analysis and modification more difficult.
• Tamper-Evident Design: Incorporate physical tamper-evident seals, tamper-proof screws, or one-way plastic clips that make unauthorized physical access immediately obvious. However, Jose acknowledges that even these measures can be bypassed by dedicated attackers, as demonstrated by the Defcon Tamper-Evident Village.

For Consumers:

Individual consumers, particularly those on a budget, are often the primary targets of this attack vector. Jose offers several practical recommendations:

• Avoid Repackaged/Pre-owned IoT Devices: The simplest defense is to avoid purchasing any "repackaged," "pre-owned," or "refurbished" IoT devices, especially from non-reputable vendors. The potential savings rarely outweigh the security risks.
• Buy from Reputable Vendors: Stick to new devices purchased directly from established manufacturers or highly reputable retailers. While not foolproof (as alleged NSA interceptions demonstrate), it significantly reduces the risk of backdoored devices.
• Question Connectivity: Not everything needs to be connected to the internet. For non-technical users, consider whether the "convenience" of an IoT device justifies the potential privacy and security risks.
• Proactive Firmware Updates (Technical Users): For technical users, proactively check for and install legitimate firmware updates directly from the vendor's site, rather than relying solely on push notifications.
• Engage in Bug Bounty Programs: Jose encourages technical individuals to participate in IoT bug bounty and vulnerability disclosure programs. He argues that vulnerabilities requiring physical access, often excluded from scope, should be included, as his research clearly demonstrates their real-world impact. Raising awareness among suppliers is crucial for driving future fixes.

Broader Implications:

Jose also considered scenarios where the attack could be more potent, such as a malicious threat actor setting up their own online store, selling backdoored devices at slightly below market value to specific targets. In this context, the attack becomes scalable and highly targeted, bypassing the fraud prevention teams of large retailers. The clash between open-source communities seeking to customize or secure devices and vendors aiming to lock down their firmware will continue, highlighting the tension between user control and vendor-controlled security. While a fully open-source ecosystem where users flash their own blank firmware offers a sustainable solution, its usability challenges make it unrealistic for the general public in the short term.

Key Takeaways

• Retailers' Failure: Retailers consistently fail to adequately sanitize returned IoT devices, allowing backdoored hardware to be resold as "repackaged" or "pre-owned."
• Firmware Persistence: Custom firmware modifications can persist through "factory resets" and superficial checks, remaining active even after devices are resold to new owners.
• Scalable Attack Vector: While difficult to execute at scale against reputable retailers, this attack vector is highly scalable for a malicious actor operating their own sales channel, potentially enabling widespread surveillance or botnet creation.
• Physical Access is Critical: Vulnerabilities requiring physical access to IoT devices have significant real-world implications, enabling sophisticated attacks like persistent surveillance (e.g., modified vacuum robot cameras) or network infiltration.
• Consumer Caution is Key: Consumers should be extremely wary of purchasing any pre-owned or repackaged IoT devices, as they carry an elevated risk of being compromised. Prioritize buying new devices from reputable vendors.
• Advocate for Stronger Security: Manufacturers must implement robust security measures like secure boot and firmware signing, and the security community should advocate for bug bounty programs to include physical access vulnerabilities to drive industry-wide improvements.

About the Speaker(s)

Matei Jose, whose full name is Mate Anthony Joseph's, is a Senior Penetration Tester at Happening XYZ, where he works with a team of cybersecurity professionals. Beyond his corporate role, Matei is also the founder of Hiveh Hack, a boutique cybersecurity company he built alongside his wife. He describes himself as someone who simply loves hacking things, whether it's websites, people, or devices, driven by a passion for exploring vulnerabilities. While he humbly states he is not a hardware hacking expert, he is deeply fascinated by the potential impact of IoT hacking, particularly how actions in a virtual environment can manifest in the physical world. His personal experiences and curiosity have driven his research into the practical implications of IoT security flaws.