OptFuzz: Optimization Path Guided Fuzzing for JavaScript JIT Compilers

OptFuzz: Optimization Path Guided Fuzzing for JavaScript JIT Compilers

Speakers: Jiming Wang, Yan Kang, Chenggang Wu, Yuhao Hu, Yuanming Lai, Mengyao Xie, Charles Zhang, Tao Li, Zhe Wang

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=Byp_XmCh_9k

Overview

This talk introduces OptFuzz, a novel fuzzing system designed to uncover vulnerabilities within JavaScript Just-In-Time (JIT) compilers by specifically targeting their optimization paths. Presented by Jiming Wang at USENIX Security '24, OptFuzz addresses a critical gap in traditional fuzzing methodologies, which often prove inefficient in thoroughly testing the complex optimization logic inherent in modern JIT engines. Given that JavaScript engines are fundamental components of web browsers, PDF readers, and numerous other applications, the security of their JIT compilers is paramount.

The core premise of OptFuzz stems from the observation that many JIT vulnerabilities arise when specific optimizations are triggered under unforeseen conditions, leading to exploitable flaws. Existing coverage-guided fuzzers, while effective for general code exploration, often fail to adequately explore the intricate execution paths within optimization routines. OptFuzz tackles this challenge by introducing the concept of Optimization Trunk Pass (OPD Pass), which serves as a refined feedback mechanism to guide fuzzing towards deeper and more comprehensive exploration of JIT optimizations. This work is crucial for bolstering the security of the ubiquitous JavaScript ecosystem, providing a more targeted and effective approach to unearthing these hard-to-find bugs.

Background

▶ Watch: Problem: Age coverage fuzzing is inefficient for JIT optimizations (0:50)

JavaScript JIT compilers are highly sophisticated components at the heart of modern JavaScript engines, responsible for dynamically compiling frequently executed JavaScript code into native machine code to achieve high performance. This complexity, however, comes at a significant security cost. JIT compilers incorporate numerous aggressive optimizations, such as Common Subexpression Elimination (CSE) and Loop-Invariant Code Motion (LICM), which transform the program's Intermediate Representation (IR). The intricate nature of these transformations, coupled with the need to maintain semantic equivalence, makes JIT compilers fertile ground for vulnerabilities. In recent years, a steady stream of high-severity bugs, often leading to arbitrary code execution, has been discovered in JIT compilers across mainstream JavaScript engines like V8 (Chrome), SpiderMonkey (Firefox), and JavaScriptCore (Safari).

A common characteristic of these vulnerabilities is their origin in an optimization being triggered under specific, often unexpected, conditions, leading to an incorrect or insecure code transformation. Traditional fuzzing techniques, particularly age coverage-guided fuzzing, aim to explore as much of the program's code as possible. In this paradigm, a test case that executes a new "age" (a basic block or edge in the Control Flow Graph) is preserved as a seed for further mutation. While effective for general code exploration, the researchers found that merely exploring new code paths is inefficient for discovering bugs specifically within JIT optimizations. The reason lies in the nuanced execution patterns of these optimizations.

The OptFuzz research meticulously analyzed JIT optimizations in mainstream JavaScript engines, leading to the identification of a critical concept: the optimization pass. An optimization pass refers to a specific code execution path through the outer loop of an optimization, leading to a location where a code transformation actually occurs. Based on this, the team summarized three key observations that highlight the limitations of existing fuzzing approaches for JIT optimization bugs:

1. Entering Optimization is Not Equivalent to Triggering Optimization: During JIT compilation, every optimization routine is executed. However, not every execution will result in an actual code transformation. Engine designers craft optimizations for specific JavaScript program patterns, meaning multiple, often complex, conditions must be met for an optimization to be truly "triggered" and perform its code transformation. Traditional fuzzing might reach the optimization code but fail to satisfy the conditions to trigger the actual transformation, thus missing potential bug vectors.

1. Age Coverage-Guided Fuzzing May Overlook Test Cases that Trigger New Optimization Passes: It was observed that the execution of certain optimization passes might inadvertently cover all the ages (basic blocks or edges) of other, distinct optimization passes due to code reuse within the JIT compiler's structure. Consequently, a test case that triggers a new and potentially vulnerability-exposing optimization pass might not be preserved as a seed by an age coverage-guided fuzzer because it doesn't reach any new ages. This can lead to critical optimization paths remaining largely untested.

1. Imbalanced Testing for Optimization Passes: The conditions required to trigger certain optimization passes are often easier to satisfy than others. This leads to an imbalance in testing, where easily satisfied paths undergo extensive fuzzing, while those with more challenging conditions—which might conceal more subtle bugs—are tested infrequently. Traditional fuzzers lack the granularity to prioritize testing of these under-explored optimization paths.

These observations collectively highlight that for effective fuzzing of JIT optimizations, it is not sufficient to merely reach the optimization code; it is crucial to comprehensively test the various execution paths through the optimization logic itself.

Key Findings

▶ Watch: Three key observations on JIT optimization fuzzing (3:00)

The central insight derived from the background observations is that when fuzzing JIT optimizations, the focus must shift from simply reaching the optimization code to comprehensively testing the specific execution paths that lead to and execute code transformations. This led to the development of OptFuzz, which uses the concept of an optimization pass as a refined feedback mechanism.

The primary contributions and key findings of the OptFuzz research are:

1. Introduction of Optimization Pass Feedback: OptFuzz proposes using optimization passes as feedback for fuzzing. If a test case executes a new optimization pass (i.e., a path within the outer loop of an optimization that leads to a code transformation), it is preserved as a seed. This ensures that the fuzzer actively seeks out and prioritizes test cases that meaningfully engage with the JIT's optimization logic, rather than just covering general code.

1. Addressing Seed Explosion with Optimization Trunk Pass (OPD Pass): A significant challenge in using "optimization pass" as feedback is the potential for seed explosion. JIT optimizations often contain nested inner loops. Different iterations of these inner loops would technically constitute different optimization passes, leading to an unmanageably large number of seeds. To counteract this, OptFuzz introduces the concept of an Optimization Trunk Pass (OPD Pass). An OPD Pass represents a path within the outer loop of an optimization, but crucially, it ignores nested inner loops. This abstraction significantly reduces the number of unique passes while still capturing the distinct high-level execution flows within an optimization. The researchers noted that an OPD Pass can also include paths that lead to an early exit branch, meaning the optimization is entered but not triggered, which is also valuable feedback.

1. Novel Seed Scheduling Strategy: To address the problem of imbalanced testing, OptFuzz implements a two-seed queue strategy. This strategy prioritizes OPD passes that have been tested less frequently. By giving these under-explored passes more opportunities for mutation and execution, OptFuzz ensures a more balanced and thorough exploration of the JIT's optimization landscape. This targeted scheduling helps uncover bugs in paths that are difficult to reach or trigger.

1. Significant Bug Discovery: Through extensive evaluation against state-of-the-art fuzzers and four mainstream JavaScript engines, OptFuzz demonstrated superior bug-finding ability. It discovered 36 new bugs, with 26 of them confirmed by vendors and resulting in the assignment of CVEs. This quantitative success underscores the effectiveness of the OPD Pass guided fuzzing methodology.

1. Enhanced Exploration of Optimization Logic: The evaluation confirmed that OptFuzz significantly outperforms other fuzzers in exploring a greater number of unique OPD passes. This directly validates the hypothesis that focusing on optimization passes as feedback leads to a deeper and more comprehensive understanding and testing of the JIT compiler's internal workings.

These findings collectively present a robust methodology for JIT compiler fuzzing, moving beyond superficial code coverage to a more profound exploration of the complex, bug-prone optimization logic.

Technical Deep Dive

▶ Watch: Introducing the 'Optimization Trunk Pass' (OPD Pass) (5:40)

The technical foundation of OptFuzz revolves around its novel feedback mechanism: the Optimization Trunk Pass (OPD Pass). Understanding this requires a closer look at how JIT optimizations are structured and how OptFuzz instruments and interprets their execution.

At its core, each JIT optimization typically contains a large outer loop. This outer loop iterates or "transverses" the program's Intermediate Representation (IR) at a specific granularity level, such as basic blocks or individual instructions. Within this outer loop, specific conditions are checked. If these conditions are met, the optimization is "triggered," meaning the program executes to a location where a code transformation occurs (e.g., modifying the IR, eliminating redundant code, or reordering operations).

An optimization pass is defined as a code execution path starting from the entry of the outer loop, proceeding through various conditional branches, and ultimately leading to a point where a code transformation takes place. For example, in a Control Flow Graph (CFG) simplification optimization, an outer loop might traverse each basic block. If a block meets specific criteria (e.g., it's empty or has a redundant jump), the optimization is triggered, and the CFG is modified. A path from the loop's start, through the condition checks, to the actual CFG modification constitutes an optimization pass.

The challenge with directly using "optimization pass" as feedback arises from nested inner loops. Many JIT optimizations feature inner loops within their outer loops. If every iteration of these inner loops were considered a distinct part of an optimization pass, the number of unique passes would explode, making the fuzzer's state space unmanageable and leading to excessive seed preservation.

To mitigate this seed explosion problem, OptFuzz introduces the Optimization Trunk Pass (OPD Pass). An OPD Pass is a simplified representation of an optimization pass. It represents a path within the outer loop, but critically, it ignores any nested inner loops. This means that regardless of how many times an inner loop iterates, or which paths are taken within that inner loop, as long as the execution path through the outer loop remains the same (up to the point of ignoring the inner loop), it's considered the same OPD Pass. This abstraction significantly reduces the number of distinct passes while still providing valuable, high-level feedback on the fuzzer's exploration of the optimization's outer logic. It's also important to note that an OPD Pass can represent paths that lead to an "early exit branch" within the outer loop, where the optimization is entered but not triggered. These "non-triggering" paths are also valuable feedback, as they represent specific conditions where an optimization could have been triggered but wasn't, which might be relevant for finding edge cases.

The OptFuzz system is built upon AFL (American Fuzzy Lop) and comprises two main modules:

1. Instrumentation Module: This module is responsible for identifying the outer loops and nested inner loops within the JIT compiler's source code. It marks the start and end points of these loops and identifies the locations where code transformations occur. This instrumentation provides the necessary hooks for tracking OPD Passes during execution. The process involves static analysis of the JIT compiler's source to identify the structural elements relevant to optimization passes.

1. Fuzzing Module: This module integrates with AFL and incorporates OptFuzz's specialized strategies based on OPD Passes:

• New Seed Preservation Strategy: If a test case executes a new OPD Pass (one not previously observed), it is preserved as a seed. This directly addresses the second observation by ensuring that test cases triggering unique optimization behaviors are retained, even if they don't necessarily cover new basic blocks.
• New Seed Scheduling Strategy: To counter the problem of imbalanced testing, OptFuzz employs a two-seed queue. This strategy prioritizes OPD passes that have been tested less frequently. Seeds associated with these under-tested passes are given more opportunities for mutation and execution, pushing the fuzzer to explore less-trodden optimization paths. The system actively tracks the number of times each OPD Pass has been executed.
• Seed Trimming Strategy: While not explicitly detailed in the transcript, seed trimming in fuzzing typically involves reducing the size of a preserved seed while maintaining its ability to trigger the desired behavior (in this case, the OPD Pass). This helps in generating smaller, more efficient test cases for subsequent mutations.

A known challenge with OPD Passes is that by ignoring inner loops, some paths within those inner loops might still be under-tested. The researchers acknowledge this and suggest that while general understanding indicates that frequent testing of outer loops often leads to comprehensive testing of inner loops, there might be specific, less-tested paths within inner loops that require more fine-grained feedback. They plan to investigate this in future work, potentially by introducing additional instrumentation or feedback mechanisms that incur minimal overhead. Another consideration is handling "early exit branches" within OPD passes. OptFuzz mitigates this by using its seed scheduling strategy, prioritizing OPD passes that have been tested less, including those that lead to early exits, ensuring that even non-triggering conditions are explored systematically.

Experimental Validation and Results

▶ Watch: Evaluation results: 36 new bugs discovered by OptFuzz (7:50)

While the talk did not present a traditional live "demo," it provided comprehensive experimental validation of OptFuzz's effectiveness against state-of-the-art fuzzers. The evaluation aimed to demonstrate OptFuzz's superior bug-finding ability and its efficiency in exploring optimization paths compared to existing methods.

The researchers conducted experiments using OptFuzz alongside four widely recognized, state-of-the-art fuzzers: SuperD, Fuzzilli, Fuzz-NG, and AFL. These fuzzers were deployed against four mainstream JavaScript engines, although the specific names of these engines were not disclosed in the transcript. This setup allowed for a direct comparison of bug discovery rates and exploration efficiency.

The results were compelling:

1. Superior Bug-Finding Ability: OptFuzz discovered a total of 36 new bugs across the tested JavaScript engines. A significant portion of these, 26 bugs, were confirmed by the respective vendors, leading to the assignment of CVEs (Common Vulnerabilities and Exposures). This quantitative outcome clearly demonstrates OptFuzz's efficacy in identifying previously unknown vulnerabilities that eluded other advanced fuzzing techniques.

1. Enhanced Exploration of OPD Passes: To evaluate OptFuzz's core mechanism, the researchers compared its ability to explore unique OPD passes against other fuzzers. The results showed that OptFuzz was able to explore more OPD passes than any of the other tested fuzzers. This directly validates the hypothesis that using optimization pass feedback significantly improves the depth and breadth of exploration within JIT optimization logic.

1. Mitigation of Seed Explosion: The researchers also conducted an experiment to compare the number of seeds generated when using "Loop Pass" (optimization paths that do not ignore inner loops) versus "OPD Pass." As predicted, using Loop Pass resulted in an explosion in the number of seeds, making it impractical for sustained fuzzing. In contrast, OptFuzz, using OPD Passes, yielded the fewest number of seeds among the various strategies tested, while simultaneously finding the most number of bugs. This crucial finding confirms that the OPD Pass abstraction effectively manages seed complexity without compromising bug discovery. The balance between a manageable seed queue and comprehensive coverage is a key strength of OptFuzz.

These experimental results provide strong evidence that OptFuzz's methodology, leveraging optimization path guided fuzzing and the OPD Pass concept, represents a significant advancement in the field of JIT compiler security testing. Its ability to discover a high number of confirmed vulnerabilities while maintaining efficient resource usage highlights its practical value.

Defensive Implications

▶ Watch: Conclusion: Summary of OptFuzz's contributions and impact (8:50)

The findings presented by OptFuzz carry significant implications for both developers of JavaScript JIT compilers and security professionals seeking to defend against or identify vulnerabilities in these critical components.

For JIT compiler developers and maintainers, OptFuzz provides a clear roadmap for more effective internal security testing:

• Prioritize Optimization Path Coverage: Developers should shift focus beyond general code coverage metrics to specifically track and ensure comprehensive coverage of optimization passes, particularly OPD Passes. This means not just checking if an optimization function is called, but verifying that all relevant internal paths leading to code transformations (and even early exits) are exercised.
• Integrate OPD Pass Feedback into CI/CD: The methodology of OptFuzz can be integrated into continuous integration and continuous deployment (CI/CD) pipelines. By instrumenting JIT compilers to collect OPD Pass feedback during internal fuzzing campaigns, developers can gain deeper insights into which optimization paths are well-tested and which remain obscure.
• Targeted Patching and Rework: When a bug is found in an optimization, the OptFuzz methodology can help identify the specific OPD Pass that triggered it. This precision allows developers to not only patch the immediate vulnerability but also to re-evaluate the testing coverage for that particular optimization path, potentially identifying other latent issues.
• Proactive Vulnerability Hunting: Developers can use the OptFuzz framework to proactively hunt for bugs, especially when implementing new optimizations or refactoring existing ones. The tool can highlight areas where conditions for triggering an optimization are complex or rarely met, indicating potential blind spots for traditional testing.

For security researchers, penetration testers, and vulnerability analysts:

• Enhanced Fuzzing Strategies: OptFuzz offers a powerful new strategy for discovering JIT vulnerabilities. Researchers can adopt or adapt the principles of OPD Pass guided fuzzing to improve their own tools and methodologies, potentially leading to the discovery of more sophisticated and subtle bugs.
• Understanding Exploit Primitives: Understanding which optimization passes are triggered by specific JavaScript code snippets can provide valuable insights into exploit primitives. If a vulnerability is tied to a particular OPD Pass, researchers can better understand the conditions required to trigger it and potentially chain it with other vulnerabilities.
• Staying Updated: Given the continuous discovery of JIT vulnerabilities, it remains paramount for users and administrators to keep their JavaScript engines (and thus, their web browsers and other applications) updated to the latest secure versions. OptFuzz contributes to this by helping vendors find and patch these critical flaws faster.

In essence, OptFuzz emphasizes that a deeper, more context-aware understanding of JIT compiler execution paths is crucial for both discovering and defending against the complex vulnerabilities inherent in these high-performance, security-critical components.

Key Takeaways

• JavaScript JIT compilers are complex and frequently exploited components, with many vulnerabilities stemming from subtle bugs in their optimization routines.
• Traditional age coverage-guided fuzzing is often inefficient for finding JIT optimization bugs because it fails to adequately explore the intricate internal execution paths of optimizations.
• The concept of an Optimization Pass (a code path within an optimization's outer loop leading to a code transformation) is critical feedback for effective JIT fuzzing.
• Optimization Trunk Pass (OPD Pass) is a novel abstraction that ignores nested inner loops, solving the "seed explosion" problem while still providing robust feedback for guiding fuzzing.
• OptFuzz, by employing OPD Pass feedback and a two-seed queue scheduling strategy, significantly improves bug discovery, having found 36 new bugs (26 confirmed with CVEs) in mainstream JavaScript engines.
• Defenders and developers should prioritize comprehensive testing of JIT optimization paths, integrating specialized fuzzing techniques like OptFuzz to proactively identify and mitigate vulnerabilities.

About the Speaker(s)

The research behind OptFuzz was a collaborative effort by a team of researchers including Jiming Wang, Yan Kang, Chenggang Wu, Yuhao Hu, Yuanming Lai, Mengyao Xie, Charles Zhang, Tao Li, and Zhe Wang. Jiming Wang presented the work at USENIX Security '24. While specific affiliations and detailed titles were not provided in the transcript for each author, the collective expertise of the team from their respective institutions (often universities and security research labs) focuses on system security, vulnerability research, and advanced fuzzing techniques, particularly in complex software components like JavaScript engines. Their work contributes significantly to advancing the state-of-the-art in automated vulnerability discovery.