Don't Waste My Efforts: Pruning Redundant Sanitizer Checks by Developer-Implemented Type Checks

Don't Waste My Efforts: Pruning Redundant Sanitizer Checks by Developer-Implemented Type Checks

Speakers: Yizhuo Zhai, Postdoctoral Researcher at Georgia Tech (formerly PhD at UC Riverside); Paul Yu, Researcher at UC Riverside / US Army Research Lab; Srikanth V. Krishnamurthy, Professor at UC Riverside / US Army Research Lab

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=6wc2cqw0S9o

Overview

Type confusion vulnerabilities continue to pose a significant threat in C++ applications, potentially leading to system crashes, denial-of-service, or even arbitrary code execution. This talk, "Don't Waste My Efforts: Pruning Redundant Sanitizer Checks by Developer-Implemented Type Checks," presented by Yizhuo Zhai, delves into an innovative approach to mitigate these critical vulnerabilities effectively and efficiently. The core of the work, a joint effort from UC Riverside and the US Army Research Lab, introduces Tunify, a tool designed to intelligently optimize the application of runtime type sanitizers.

Tunify addresses a fundamental dilemma in C++ security: the trade-off between comprehensive protection and performance overhead. While existing sanitizer-based solutions offer full protection against type confusion, their performance impact can be prohibitive for large, performance-critical software. Conversely, developer-implemented type checks, though lightweight, are often incomplete and inconsistent. This research bridges this gap by automatically identifying and leveraging developer-implemented custom runtime type information (RTTI) to prune redundant sanitizer checks, thereby achieving full protection with significantly reduced overhead. The insights and methodology presented are crucial for developers, security engineers, and researchers working with C++ codebases, offering a path to more secure and performant applications.

Background

▶ Watch: Introduction to type confusion and unsafe downcasting (0:00)

Type confusion arises in C++ when an object intended to be of one type (e.g., Type A) is mistakenly treated as an incompatible type (Type B). This often occurs during type casting, particularly downcasting, where a pointer to a base class is cast to a pointer of a derived class. For instance, considering a Shape base class with Circle and Square derived classes: if a Shape pointer that actually points to a Square object is downcast to a Circle pointer, attempting to access a radius field (unique to Circle) would lead to an invalid memory access because that memory region does not correspond to a radius in the Square object's layout. Such an access can corrupt memory, crash the program, or, in severe cases, enable arbitrary code execution.

Existing dynamic mitigations for type confusion each present their own set of advantages and limitations:

1. C++ dynamic_cast Operations: The C++ language itself offers dynamic_cast for safe downcasting. These operations perform runtime type checks by extracting type information, typically through the virtual function table (vtable), and verifying compatibility. However, dynamic_cast has two significant limitations. First, the base class of the derived type must possess a virtual table, meaning at least one virtual function must be defined. Second, the process of traversing the linked list of type information to determine compatibility introduces considerable overhead, making it impractical for performance-critical software like web browsers (e.g., Chrome). Consequently, dynamic_cast cannot provide full protection across all scenarios and often incurs too high a performance penalty.

1. Sanitizer Approaches (e.g., HexType): To overcome the limitations of dynamic_cast, sanitizer-based approaches were developed. These techniques encode compatible type information into metadata associated with an object at its allocation site, often organizing this data in efficient lookup structures like binary trees. During compilation, sanitizer checks are inserted before each downcast operation. At runtime, these checks query the object's metadata to ensure type compatibility. This method provides full protection against type confusion vulnerabilities. However, the frequent execution of these sanitizer checks throughout the program's execution introduces a substantial performance overhead, making it challenging to deploy in production environments where every millisecond counts. For example, the HexType sanitizer, while effective, can impose significant runtime costs.

1. Custom Runtime Type Information (RTTI) / Developer-Implemented Type Checks: Many developers, aware of type safety concerns and the overhead of dynamic_cast, implement their own lightweight type checks. A common pattern involves adding a type field to a base class, initializing it with a specific value in the constructor of each derived class, and then checking this field before performing a downcast. This approach is highly efficient, typically involving just a memory read and a comparison, resulting in low overhead. The critical drawback, however, is that this protection is entirely dependent on developer diligence. There's no guarantee that developers will consistently define such type information fields, nor that they will insert checks before every necessary cast. Therefore, custom RTTI cannot provide full, systematic protection.

The fundamental problem addressed by Zhai and his colleagues is how to combine the comprehensive security of sanitizer approaches with the low overhead of developer-implemented type checks. This work seeks to leverage the best aspects of both worlds, providing robust protection without crippling performance.

Key Findings

▶ Watch: Identifying the gap and proposing Tunify's combined approach (5:00)

The research behind Tunify yielded several crucial findings that underpin its effectiveness and highlight the practical relevance of its approach:

• Prevalence of Custom RTTI in Large-Scale Software: A systematic investigation into seven popular, large-scale C++ software projects revealed that six of them extensively utilize custom RTTI (Runtime Type Information). This finding underscores that developer-implemented type checks are not an isolated phenomenon but a widespread, albeit informal, practice for managing type safety. This prevalence indicates a significant opportunity to leverage existing developer efforts for more robust and efficient security.

• Effective Identification of Safe Casts: Tunify demonstrated a strong capability to automatically identify "safe casts"—those downcast operations already protected by developer-implemented type checks. By precisely determining which casts are covered by these lightweight checks, Tunify can prevent the insertion of redundant, high-overhead sanitizer checks. While the absolute number of identified safe downcasts might appear small compared to the total number of downcasts in a program, their strategic identification proves vital for performance optimization.

• Substantial Overhead Reduction: The most compelling finding is the significant reduction in performance overhead achieved by Tunify compared to traditional sanitizer approaches. The evaluation showed that Tunify can reduce the number of sanitizer checks by a remarkable margin, leading to overall overhead reductions ranging from 25% to 75% for large-scale C++ software. A notable example cited was the Speedometer benchmark running on Chromium: while the HexType sanitizer performed thousands of millions of checks, Tunify reduced this to only 241 million checks, representing an 84% reduction in sanitizer operations. This demonstrates that Tunify successfully bridges the gap between full type confusion protection and acceptable performance, making comprehensive runtime security a more viable option for production systems.

Technical Deep Dive

▶ Watch: Tunify's two-step approach: inference and verification (6:00)

The Tunify approach is a sophisticated, multi-stage process designed to intelligently prune redundant sanitizer checks by leveraging developer-implemented type checks. It operates by first understanding how developers encode type information and then using that understanding to statically verify the safety of downcast operations.

The overall Tunify workflow consists of three main steps:

1. Custom Type Information Inference: This initial step involves systematically analyzing the program's source code to identify and collect all classes, organize them into a type hierarchy, and, crucially, infer how developers store and manage custom runtime type information (RTTI).
2. Static Safe Casting Verification: Once custom RTTI patterns are inferred, Tunify compiles the source code into LLVM Intermediate Representation (IR). It then uses the inferred type information in conjunction with control flow analysis to identify downcasts that are already adequately protected by developer-implemented checks. These are termed "safe casts."
3. Lightweight Instrument Program Generation: For casts identified as safe, Tunify omits the insertion of heavyweight sanitizer checks. For all other casts that are not covered by developer checks, it falls back to the robust sanitizer approach. This selective instrumentation results in a program that retains full protection against type confusion while significantly reducing runtime overhead.

Let's delve deeper into the first two technical steps:

Custom RTTI Inference

The primary challenge in inferring custom RTTI lies in its highly customized nature. Developers employ diverse patterns, lacking a standard, documented, or formal structure. To address this, Tunify systematically investigates large C++ programs and categorizes these patterns into three common types:

1. RTTI Encoded in a Base Class Field: This is a very common pattern, exemplified by classes within Chromium. In this setup, a base class (e.g., Shape) declares a field (e.g., class_type). Derived classes (Circle, Square) then pass specific, distinct enumeration constants or integral values to the base class constructor to initialize this class_type field, thereby identifying their concrete type. The base class often provides utility functions (e.g., get_type()) to access this type identifier. Tunify identifies this pattern by looking for enumeration constants, checking if they are assigned to a field in the constructor, and verifying that these fields, once initialized, are not subsequently changed.

1. RTTI as a Constant Returned by a Virtual Method: In this category, instead of an explicit field, the base class declares a virtual function (e.g., get_shape_type()). Each derived class then overrides this virtual function to return a unique constant value (e.g., SHAPE_TYPE_CIRCLE, SHAPE_TYPE_SQUARE) that distinguishes its type. Tunify infers this pattern by collecting enumeration constants and checking if they are returned by virtual methods in derived classes, ensuring these return values are unique across the hierarchy.

1. RTTI Defined as a Type Check Function: This pattern involves the base class declaring a set of virtual boolean functions (e.g., is_circle(), is_square()), all initially returning false. Each derived class then overrides only the virtual function corresponding to its own type to return true. For example, the Circle class would override is_circle() to return true, while is_square() would still return false. Tunify identifies this by examining derived classes for overridden virtual functions that return a unique true value, ensuring that only one such function returns true for a given type.

For the first two categories, the inference process checks if collected enumeration constants are assigned to fields in constructors or returned by virtual methods, and critically, if they remain constant post-initialization. For the third category, Tunify verifies if derived classes override virtual functions to return unique true values, ensuring a clear distinction between types.

Static Safe Casting Verification

After inferring the custom RTTI patterns, Tunify proceeds to statically verify downcast operations. This stage operates at the LLVM IR level, which provides a rich intermediate representation suitable for detailed static analysis.

The core logic here involves:

1. Identifying Comparison Operations: Tunify scans the LLVM IR for comparison operations, particularly those embedded within if statements or switch cases. These are the typical constructs developers use to implement their custom type checks.
2. Inferring Correct Type: When a comparison operation is identified that matches one of the inferred custom RTTI patterns (e.g., if (object->get_type() == SHAPE_TYPE_CIRCLE)), Tunify uses the previously collected type information to infer the concrete type of the object at that program point.
3. Dominance Analysis: With the concrete type inferred, Tunify then performs control flow analysis, specifically looking for downcast operations that are dominated by these developer-implemented type checks. A cast is considered "dominated" if it can only be reached after the type check has successfully executed. Furthermore, Tunify ensures that there are no intervening operations between the type check and the cast that could alter the object's type or invalidate the type check's premise.
4. Identifying Safe Casts: Any downcast operation that satisfies these conditions (dominated by a valid developer type check with no intervening type-altering operations) is classified as a "safe cast." For these safe casts, Tunify can confidently omit the insertion of an additional runtime sanitizer check, knowing that the developer's logic already provides the necessary type safety.

Implementation Details

The implementation of Tunify leverages established compiler infrastructure. For the initial parsing of type information and understanding the C++ abstract syntax tree, Tunify utilizes libclang. The subsequent stages, including LLVM IR processing, control flow analysis, and instrumentation, are built upon LLVM version 14.0.5. The entire codebase for Tunify comprises over 9,000 lines of code. For comparative evaluation, the HexType sanitizer, a prominent type confusion mitigation, was also updated to LLVM version 14.0.5 to ensure a fair performance comparison.

Demo / Proof of Concept

▶ Watch: How static safe casting verification identifies protected casts (8:40)

The talk primarily focused on the technical methodology and comprehensive evaluation results of Tunify rather than a live demonstration or a specific proof-of-concept exploit against a vulnerable application. The effectiveness of Tunify was showcased through quantitative metrics on large-scale C++ software, illustrating its ability to reduce sanitizer overhead in real-world scenarios.

Defensive Implications

▶ Watch: Transition to implementation details (10:00)

The Tunify research offers several crucial implications for developers, security engineers, and organizations working with C++ codebases:

1. Optimized Type Confusion Mitigation: The most direct implication is the availability of a methodology and tool (Tunify, once open-sourced) that can provide full protection against type confusion vulnerabilities with significantly reduced performance overhead. Defenders no longer have to choose between robust security and application performance to the same extent. Integrating Tunify into compiler toolchains or build processes could become a standard practice for C++ projects, especially those with stringent performance requirements.

1. Leveraging Developer Intent: Tunify highlights the value of recognizing and leveraging developer-implemented custom RTTI. Instead of viewing these as informal or incomplete solutions, this work demonstrates how they can be systematically analyzed and integrated into a broader security strategy. This encourages developers to continue employing clear, consistent custom RTTI patterns, as their efforts can now directly contribute to performance optimization alongside security.

1. Enhanced Compiler Toolchains: The research showcases the potential for more intelligent compiler toolchains. By understanding code semantics, control flow, and developer-specific patterns, compilers can make smarter decisions about where to insert security instrumentation. This moves towards a future where security features are seamlessly integrated and optimized, reducing the burden on developers.

1. Informed Security Decisions: Security teams can use the insights from Tunify to make more informed decisions about type confusion mitigations. Understanding the prevalence of custom RTTI and the potential for overhead reduction can help in assessing existing codebases, prioritizing remediation efforts, and evaluating the feasibility of deploying comprehensive runtime protections.

1. Addressing a Persistent Threat: Type confusion remains a critical vulnerability class, frequently exploited in the wild. Tunify provides a practical and scalable solution that can make a substantial impact on the security posture of C++ applications, reducing the attack surface without imposing an unacceptable performance penalty. This empowers organizations to deploy stronger defenses against a persistent and dangerous class of software flaws.

Key Takeaways

• Type confusion is a prevalent and severe vulnerability in C++ applications, potentially leading to crashes or arbitrary code execution.
• Traditional sanitizer-based solutions offer full protection against type confusion but incur high performance overhead, making them impractical for many large-scale applications.
• Developers frequently implement their own custom Runtime Type Information (RTTI) checks, which are lightweight but offer incomplete protection.
• Tunify is an automated tool that intelligently combines the strengths of both approaches by identifying and leveraging developer-implemented custom RTTI to prune redundant sanitizer checks.
• This novel methodology achieves full type confusion protection with significant performance gains, reducing sanitizer overhead by 25% to 75% for large C++ software, including an 84% reduction in checks on the Speedometer benchmark with Chromium.
• Tunify represents a crucial advancement towards practical, high-performance runtime type confusion mitigation, allowing for robust security without prohibitive performance costs.

About the Speaker(s)

The research presented in this talk is a collaborative effort by Yizhuo Zhai, Paul Yu, and Srikanth V. Krishnamurthy. Yizhuo Zhai, the primary presenter, is currently a Postdoctoral Researcher at Georgia Tech, where he works with Professor Tesu Kim. This work represents his final research contribution during his PhD studies at UC Riverside, conducted under the guidance of "awesome professors." Paul Yu and Srikanth V. Krishnamurthy are co-authors on this paper, contributing from their affiliations with UC Riverside and the US Army Research Lab. Their joint work highlights a strong collaboration between academic research and governmental defense initiatives, aiming to enhance the security and performance of critical software systems.