What is Dead May Never Die: The Immortality of SDK Bugs

What is Dead May Never Die: The Immortality of SDK Bugs

Speakers: Richard Lawshae (Principal Security Researcher, Ksite Technologies)

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=MTZ9_IfZ7Bk

Overview

In "What is Dead May Never Die: The Immortality of SDK Bugs," Richard Lawshae, a Principal Security Researcher at Ksite Technologies, delves into the pervasive and enduring threat posed by vulnerabilities within Software Development Kits (SDKs) used in network chipsets. Lawshae, also known as Ricky Lashe or Headless Lique, highlights how these bugs, often introduced early in the development lifecycle, can persist for years, even decades, across a vast and fragmented ecosystem of devices, making them a significant concern for IoT security.

The talk frames SDKs not just as software toolkits, but as "system development kits" encompassing a broad array of applications, drivers, scripts, toolchains, and example code necessary for integrating a chip into a third-party product. Lawshae's research focuses on network chipsets due to their inherent exposure to remote exploitation, making them prime targets for botnet authors and other malicious actors. The core thesis is that the complex supply chain, coupled with development shortcuts and a lack of rigorous security auditing, imbues these vulnerabilities with an almost immortal quality, allowing them to resurface and be exploited long after their initial discovery.

This presentation is crucial for device manufacturers, security researchers, and anyone involved in the IoT supply chain. It sheds light on the often-overlooked security debt accumulated through the widespread adoption of unhardened SDK components. Lawshae articulates how this problem is compounded by frequent corporate acquisitions, which lead to companies inheriting product lines and their associated SDKs without full knowledge of their internal workings or security posture, creating a fertile ground for long-lived, high-impact vulnerabilities.

Background

▶ Watch: Introduction: What is Dead May Never Die (0:00)

The proliferation of complex System on Chip (SoC) designs has made it increasingly challenging for device manufacturers to integrate new chipsets efficiently. Chipset vendors address this by providing Software Development Kits (SDKs), comprehensive bundles of tools, drivers, libraries, and example code designed to streamline the integration process, minimize development time, and keep costs down. While intended to accelerate innovation, this convenience often comes at a significant security cost.

Several factors contribute to the "immortality" of SDK bugs. Firstly, the dynamic landscape of corporate acquisitions frequently sees chipset lines change hands. New owners often inherit SDKs they weren't involved in developing, leading to a lack of deep understanding of their security implications or hidden functionalities. Lawshae recounts an incident during an audit of an Oculus device, where an unknown UDP service, originating from a mobile chipset maker's SDK, was discovered. This service allowed specific packet strings to knock the device offline, highlighting how features from upstream vendors can persist unbeknownst to the final product manufacturer.

Secondly, the pervasive use of example code supplied within SDKs presents a critical vulnerability vector. While never intended for production, device makers frequently adopt this code to expedite development. This example code is typically unaudited, unhardened, and often riddled with security flaws, effectively embedding vulnerabilities directly into final products.

Thirdly, SDKs often rely on numerous third-party and open-source libraries. Maintaining these dependencies and patching against newly discovered vulnerabilities is a monumental task, frequently neglected. Furthermore, forks of open-source projects within SDKs can diverge from their original upstream, failing to incorporate critical security updates. This creates a fragmented and vulnerable ecosystem where outdated components become persistent attack surfaces.

Finally, the scattered install base magnifies the problem. Once a chip, and its underlying SDK, is integrated into a product, that product is then distributed across countless devices by various vendors. A vulnerability discovered in one instance can rapidly be found across an entire product line or even across different vendors using the same chip. As Lawshae aptly puts it, "you can't put the genie back in the tube of toothpaste" once a vulnerability is out in the wild, particularly with proprietary services that are often copy-pasted across different SDK versions. The speaker specifically targets network chipsets, as their inherent connectivity makes them susceptible to remote exploitation, with command injection bugs being a particular favorite due to their ability to exploit diverse architectures from a single vulnerability.

Key Findings

▶ Watch: Oculus vulnerability example from hidden SDK service (3:00)

Lawshae's research uncovers several critical findings regarding the nature and persistence of SDK vulnerabilities. The most striking discovery is the extraordinary longevity of these bugs. He cites a Realtek UPnP implementation vulnerability he discovered in 2014 that only made it onto the CISA Known Exploited Vulnerabilities (KEV) list in 2023, a decade later, because it was still being actively exploited in the wild. This illustrates the "immortality" principle: once an SDK bug is released, it can persist and remain exploitable for an extensive period due to the widespread, unpatched deployment of affected chips.

Another key finding is the significant role of corporate acquisitions in perpetuating vulnerabilities. The complex histories of companies like Broadcom, Avago, Infinion, Cypress, and Brocade demonstrate how chip lines and their associated SDKs change ownership frequently. This transfer of responsibility often means new owners lack the institutional knowledge to identify or remediate existing flaws, effectively inheriting security debt. This dynamic makes vulnerability disclosure and coordinated patching efforts exceptionally challenging, as the original developers may no longer exist or be responsible.

Lawshae emphasizes that proprietary services and example code are primary sources of these persistent vulnerabilities. Chipmakers often copy-paste internal proprietary network protocols or utility services across different SDK versions, ensuring that any flaw in these components propagates widely. Similarly, the unhardened example code, intended for development, frequently finds its way into production devices, introducing known or easily discoverable vulnerabilities into consumer products. The speaker's personal experience with an unknown UDP service on an Oculus device, originating from a mobile chipset maker, serves as a concrete example of these hidden, vulnerable services.

Furthermore, the talk highlights that common web server components and network protocols integrated into SDKs are frequently misconfigured or poorly implemented, creating broad attack surfaces. For instance, Broadcom's HTTP attack surface often utilizes a "mashup" of microHTTP, miniHTTP, and Go-Ahead servers. Their UPnP implementation is described as "super super buggy" and is even suspected of being a ripped-off version of Ralink's implementation, suggesting a lack of original security scrutiny.

Finally, Lawshae underscores the particular danger of command injection vulnerabilities in SDKs. Due to the diverse architectures and memory layouts supported by a single SDK, a single command injection flaw can allow an attacker to exploit an entire install base. This makes them highly attractive to botnet authors and explains their high impact across the IoT landscape. The remote RPC service in Quantenna's QCS API, which exposes hundreds of Wi-Fi configuration commands, presents a prime example of how such vulnerabilities can be remotely triggered.

Technical Deep Dive

▶ Watch: Unhardened SDK example code used in production (4:00)

The technical core of Lawshae's presentation focuses on specific chipmakers and their common attack surfaces, illustrating how SDK design and corporate history contribute to enduring vulnerabilities.

One of the most complex examples is the lineage of Broadcom, which has undergone a series of significant acquisitions and name changes, involving Avago, Infinion, Cypress, and Brocade. This convoluted history means that Broadcom chipsets, widely used across various devices, carry a legacy of potentially unpatched code. Lawshae points out that Broadcom's HTTP attack surface often comprises a mix of embedded web servers like microHTTP, miniHTTP, and the ubiquitous Go-Ahead web server. The presence of multiple, often customized, HTTP stacks can introduce inconsistencies and unique vulnerabilities. Critically, their UPnP implementation is highlighted as "super super buggy" and shows evidence of being directly ripped off from Ralink's implementation. This suggests a lack of original development and security hardening, leading to inherited vulnerabilities. The speaker also mentions the YZ SDK, which was part of Broadcom's IoT portfolio before being acquired by Cypress in 2016, further fragmenting the responsibility for its security.

The enduring nature of these flaws is starkly illustrated by the Realtek UPnP vulnerability. Discovered by Lawshae in 2014, this bug was still being actively exploited nearly a decade later, leading to its inclusion on the CISA KEV list in 2023. This specific vulnerability likely resides within Realtek's proprietary UPnP service, which, as with many SDK components, is widely distributed and rarely updated by end-device manufacturers. The long exploitation window underscores the "fire and forget" nature of many SDK deployments, where initial security audits are either non-existent or insufficient, and subsequent patching is neglected.

Another significant example is the Quantenna chipset, formerly a popular choice for AT&T set-top boxes and modems, now owned by ON Semiconductor. Quantenna's SDK includes a service called QCS API, which provides "hundreds of different commands" for configuring Wi-Fi functionalities. Crucially, this service exposes an RPC service, allowing these configuration commands to be sent and executed remotely. This remote accessibility, coupled with the sheer volume of available commands, creates an extensive attack surface. If any of these commands are susceptible to input validation issues, particularly command injection, it could lead to severe consequences, including full device compromise. The speaker noted that while he wanted to delve into MediaTek and Microchip SDKs, time constraints prevented a detailed discussion, implying similar vulnerabilities exist within their ecosystems.

Lawshae also draws attention to the difficulty of finding source code for many of these SDKs, which impedes independent security research and auditing. He attempts to provide links to available code repositories where possible, recognizing that access to source code is paramount for understanding and addressing these deep-seated vulnerabilities. The recurring theme is that proprietary services, often developed with minimal security oversight and then copied across numerous SDK versions, represent a significant and persistent threat vector.

Demo / Proof of Concept

▶ Watch: Why command injection is a favorite SDK bug (6:30)

While the live talk was unfortunately cut short, preventing a detailed demonstration of exploitation, Lawshae provided compelling conceptual proofs of concept and referred to real-world impact. The primary example highlighted was the Realtek UPnP vulnerability he discovered in 2014. This bug's longevity, evidenced by its active exploitation in the wild up to 2023 and subsequent inclusion in the CISA KEV list, serves as a powerful testament to the real-world exploitability and persistence of SDK flaws. Although the specific exploit chain wasn't detailed, its presence on a critical government vulnerability list signifies its proven effectiveness in compromising affected devices.

Another illustrative example came from Lawshae's audit of an Oculus device. During this engagement, he discovered a hidden UDP service that the device manufacturer was unaware of, inherited from an underlying mobile chipset SDK. By sending a "certain string of packets" to this service, Lawshae was able to reliably knock the Oculus device offline. This demonstrates a practical proof of concept for a denial-of-service attack, originating from an unknown and unmanaged SDK component. It underscores how unexpected services can expose devices to simple yet effective attacks.

Furthermore, the discussion around Quantenna's QCS API and its RPC service inherently describes a potent attack surface. The ability to remotely send "hundreds of different commands" for Wi-Fi configuration via this RPC interface strongly implies that a well-crafted input, particularly one leveraging command injection or other input validation flaws, could lead to arbitrary code execution or significant device misconfiguration. While not a live demo, the architectural description itself functions as a strong conceptual proof of concept for remote exploitation. The speaker's emphasis on command injection as his favorite bug class for SDKs further suggests that many of these vulnerabilities would be demonstrated through injecting malicious commands into poorly sanitized input fields of these proprietary services.

Defensive Implications

▶ Watch: Broadcom/Brocade/Cypress/Infinion SDK attack surface analysis (7:50)

The "immortality of SDK bugs" presents a severe challenge for defenders, necessitating a multi-pronged approach that addresses the entire supply chain.

1. Rigorous SDK Component Auditing: Device manufacturers must treat every component of an SDK, whether proprietary or open-source, as a potential attack surface. This includes performing thorough security audits of all drivers, libraries, applications, and especially proprietary network services (like Quantenna's QCS API or the undisclosed Oculus UDP service). Relying solely on the chip vendor's assurances is insufficient.
2. Avoid Production Use of Example Code: The practice of integrating unhardened example code directly into production devices must cease. Any code derived from SDK examples should undergo the same rigorous security review, hardening, and testing as internally developed code, including threat modeling and penetration testing.
3. Proactive Dependency Management and Patching: Manufacturers must maintain an accurate inventory of all libraries and dependencies within their SDKs. This includes tracking upstream versions for open-source components and actively monitoring for security advisories. A robust patching strategy is essential to address vulnerabilities promptly, as demonstrated by the Realtek UPnP bug's decade-long exploitation window.
4. Supply Chain Transparency and Accountability: When acquiring chipsets or companies, due diligence must extend to the security posture of inherited SDKs. Contracts with chip vendors should include clauses for vulnerability disclosure, remediation commitments, and long-term support. The fragmented ownership history of entities like Broadcom/Cypress highlights the difficulty but necessity of this.
5. Network Segmentation and Least Privilege: For deployed devices, network segmentation can limit the blast radius of an exploited SDK bug. Restricting network access to proprietary services to only necessary interfaces and subnets can mitigate remote exploitation. Implementing firewall rules to block or limit access to known vulnerable ports or protocols (e.g., specific UPnP services) is a crucial interim measure.
6. Input Validation and Sanitization: Developers working with SDKs must implement robust input validation and sanitization for all user-controlled data, especially when interacting with underlying system commands. This is critical for preventing command injection vulnerabilities, which Lawshae identified as highly effective across diverse architectures.
7. Consider Built-in Security Features: Leverage any hardware-backed security features offered by the chipset, such as secure boot, hardware-rooted trust, or memory protection units, to mitigate the impact of software vulnerabilities within the SDK.
8. Vulnerability Disclosure and Coordination: Establish clear channels for security researchers to report vulnerabilities. Be prepared for complex disclosure processes involving multiple vendors in the supply chain, as was the case with the Oculus incident involving a mobile chipset maker.

By adopting these defensive strategies, device manufacturers can significantly reduce the attack surface presented by SDKs and break the cycle of "immortality" for these deeply embedded vulnerabilities.

Key Takeaways

• SDK bugs possess extraordinary longevity: Vulnerabilities within SDKs can persist for a decade or more, actively exploited in the wild, due to widespread deployment and neglected patching, as exemplified by the Realtek UPnP vulnerability.
• Supply chain complexity exacerbates the problem: Frequent corporate acquisitions and the inheritance of undocumented or poorly understood SDKs (e.g., Broadcom's history) make vulnerability tracking, disclosure, and remediation exceptionally difficult across the fragmented IoT ecosystem.
• "Example code" is a critical vulnerability source: Unhardened and unaudited example code provided in SDKs is often adopted directly into production devices, introducing easily exploitable flaws into final products.
• Proprietary services hide persistent threats: Custom, undocumented network services and protocols within SDKs are frequently copied across product lines, creating broad and often overlooked attack surfaces that can lead to remote exploitation (e.g., Quantenna's QCS API, Oculus's hidden UDP service).
• Command injection remains a potent attack vector: Due to the architectural diversity supported by SDKs, a single command injection vulnerability can be leveraged to compromise a vast array of devices, making it a favorite for botnet authors.
• Rigorous auditing and proactive management are essential: Device manufacturers must perform thorough security audits of all third-party SDK components, manage dependencies diligently, and implement robust patching strategies to mitigate the pervasive risks posed by these embedded vulnerabilities.

About the Speaker(s)

Richard Lawshae, who also goes by Ricky Lashe or Headless Lique, is a Principal Security Researcher at Ksite Technologies, based out of Austin, Texas. His primary focus lies in offensive research targeting IoT devices, where he has a track record of discovering vulnerabilities in products from notable companies such as Meta, Mazda, and Crestron. Lawshae is a seasoned conference speaker, having presented at various security events including Ruxcon, Recon, and Insomniah Hack. This talk marked his seventh appearance at DEF CON, with five presentations at Defcon proper and two at the IoT Village, highlighting his deep expertise and contributions to the security community.