Shadowed Realities: An Investigation of UI Attacks in WebXR

Shadowed Realities: An Investigation of UI Attacks in WebXR

Speakers: Chandrika Mukherjee

Conference: USENIX Security

YouTube: https://www.youtube.com/watch?v=iTR7-QhuATc

Overview

In an increasingly immersive digital landscape, Extended Reality (XR) technologies are rapidly expanding their footprint across diverse sectors, from retail and healthcare to education and entertainment. While dedicated hardware and SDKs like Meta XR Core SDK and Mixed Reality Toolkit facilitate app development for specific headsets, the emergence of WebXR offers a unified, browser-based approach to deliver XR experiences. This talk, "Shadowed Realities: An Investigation of UI Attacks in WebXR," presented by Chandrika Mukherjee of Purdue University, delves into the critical security implications of WebXR's unique architecture, particularly concerning user interface (UI) vulnerabilities.

The core of the research highlights how fundamental differences between WebXR and traditional web environments, such as the absence of iframes and the same-origin policy, coupled with security-sensitive UI properties like object overlapping, transparency, and synthetic input, create fertile ground for malicious exploitation. These vulnerabilities can be leveraged to implement dark patterns within WebXR advertising ecosystems, leading to unintended user actions, sensitive data leakage, or even malware installation. Mukherjee's work not only categorizes existing and novel UI-based attacks but also quantitatively assesses their impact on user experience through a rigorous user study, providing crucial insights for developers and platform owners to fortify WebXR security.

Background

▶ Watch: WebXR's key differences and UI properties (2:20)

Extended Reality (XR) is an umbrella term encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR), aiming to blend real and virtual worlds. Its rapid adoption is driven by major tech companies like Meta, Microsoft, and Apple, each offering proprietary headsets and development frameworks. However, the fragmentation across these platforms has spurred the growth of WebXR, a standardized framework that enables XR experiences directly within web browsers on head-mounted displays (HMDs) like MetaQuest or HoloLens. This eliminates the need for standalone application installations, allowing users to access immersive content simply by navigating to a URL and entering 3D mode. WebXR leverages existing web technologies and is supported by popular browsers and development libraries such as A-Frame, 3JS, and BabylonJS, making XR content creation more accessible.

Despite its similarities to the standard web, WebXR introduces critical architectural distinctions that pose significant security challenges. Unlike traditional web pages, WebXR lacks an equivalent of HTML iframes, which are crucial for isolating third-party content (e.g., advertisements) and enforcing security mechanisms like the same-origin policy. This policy typically prevents scripts from one origin from accessing resources from another, safeguarding user data. In WebXR, the absence of such isolation means that content from different origins can interact more freely within the shared 3D scene. Furthermore, WebXR relies on "security-sensitive UI properties" to enhance immersion, including overlapping objects for complex scene architectures, transparency for visual effects like shadows or glass, and synthetic input for dynamic, non-user-triggered interactions. While beneficial for user experience, these properties, when manipulated by malicious developers or ad service providers, can facilitate dark patterns—deceptive UI designs that coerce users into performing unintended actions, such as clicking hidden ads or disclosing sensitive information. This foundational difference in security architecture, combined with exploitable UI elements, forms the basis for the research presented, which aims to understand how these properties contribute to dark patterns and their impact on user interaction within WebXR.

Key Findings

▶ Watch: Introducing research questions and attack taxonomy (5:15)

The research makes several significant contributions to understanding UI attacks in the WebXR advertising ecosystem. Chandrika Mukherjee and her collaborators first conducted a comprehensive literature review, identifying nine prior UI-based attacks relevant to WebXR, including cursor tracking, blind spot tracking, and denial of service attacks. Building upon this, they developed a novel four-category taxonomy to classify UI-based attacks based on the primary objectives of malicious actors:

1. Click Manipulation: Attacks designed to generate revenue through deceptive ad clicks.
2. Peripheral Exploitation: Attacks that leverage blind spots or areas outside the user's immediate focus to inflate ad impressions or clicks.
3. Functionality Disruption: Attacks aimed at preventing users from completing their intended actions within an application.
4. UI-based Privacy Leakage: Attacks used to extract sensitive user information through UI manipulation.

Within this taxonomy, the researchers proposed five novel UI-based attacks specific to the WebXR environment:

• Visual Overlapping and Sequential Rendering under Click Manipulation.
• Malvertising under Peripheral Exploitation.
• GUI Switch Attack under UI-based Privacy Leakage.
• Do Overriding Attack under Functionality Disruption.

Beyond classification, the study identified 14 contributing security-sensitive UI properties that can be exploited to facilitate these attacks. To rigorously assess the impact of these attack categories on user experience, a between-subjects user study was designed and executed with 100 participants, all over 18, recruited from the university. Participants were divided into five groups (four attack categories plus one control group), with each group further split across four distinct app types: reading, gaming, shopping, and travel. These apps were chosen to represent varying contexts and interaction demands in WebXR.

The user study yielded critical findings:

• Stealthiness of Attacks: Most attack categories, particularly Click Manipulation, Peripheral Exploitation, and UI-based Privacy Leakage, largely went unnoticed by users. Their reported experience was similar to that of the control group, indicating the subtle and deceptive nature of these dark patterns. Functionality disruption was the only category consistently perceived by users.
• Impact on Engagement: While user "presence" (focus on the task) was primarily influenced by the type of app, not the attack category, the attacks demonstrably forced users to shift their engagement with the given task.
• Effectiveness of Attacks: The attacks were highly effective in achieving their malicious objectives. The malicious attention metric showed a significant level of unintended clicks by users, confirming the success of click manipulation and other attacks relying on misdirection. Similarly, the blind spot rendering fraction revealed a significant difference between peripheral exploitation scenarios and the control group, validating the efficacy of inflating ad impressions outside the user's field of view.
• Generalizability: The attacks were found to happen consistently across different application types (reading, gaming, shopping, travel) with varying levels of user presence, suggesting their broad generalizability across diverse WebXR experiences.

These findings underscore the potent threat posed by UI-based attacks in WebXR, which can compromise user privacy, facilitate financial fraud, and undermine trust in immersive digital environments, often without the user's explicit awareness.

Technical Deep Dive

▶ Watch: Five novel UI-based attacks proposed (7:00)

The technical core of this research lies in the detailed design of the five novel UI-based attacks and the development of a robust user study framework to measure their impact. The attacks exploit fundamental WebXR properties that enhance immersion but lack traditional web security safeguards.

Let's delve into two prominent novel attacks:

1. Visual Overlapping (Click Manipulation): This attack targets WebXR's ability to render objects in a 3D space, allowing them to overlap. The malicious actor, typically a developer, strategically places a clickable advertisement behind an interactive, but ultimately "unclickable," bait object. For instance, in a target shooting game, a user might see a target (the bait object) but an ad is hidden directly behind it. When the user attempts to click the visible target, the unc clickable bait object allows the click to pass through to the underlying, hidden advertisement. To maintain the illusion that the bait object is functional, synthetic input is used. This means that even though the user's direct click is intercepted by the ad, the bait object still appears to react as if clicked, perhaps animating or changing state, thereby keeping the user unaware of the misdirection.

1. Malvertising (Peripheral Exploitation): This attack, attributed to a malicious advertiser, leverages transparency and the user's limited field of view within the 3D scene. The advertiser provides an ad that is either fully or partially transparent. The user, engaging with the 3D environment, sees the intended content (e.g., another game object or a legitimate ad from a competitor) through the transparent malicious ad. When the user attempts to interact with the visible content, their click is intercepted by the transparent malicious ad. Crucially, this intercepted click can then trigger a redirect to a web page in the auxiliary screen of the HMD's browser. Since the user's primary focus is on the immersive 3D scene, they remain unaware of this background redirection, leading to inflated ad impressions or even drive-by downloads without their knowledge.

The research also briefly touches upon:

• Sequential Rendering (Click Manipulation): Implies exploiting the order in which objects are drawn to manipulate click events.
• GUI Switch Attack (UI-based Privacy Leakage): Suggests manipulating UI elements to trick users into revealing sensitive information.
• Do Overriding Attack (Functionality Disruption): Points to attacks that prevent users from completing intended tasks by overriding their interactions.

To quantify the impact of these and other attacks, the team developed a sophisticated user study framework built on A-frame and 3JS. This framework comprised a logging framework, interaction metrics, and a suite of applications.

The logging framework was designed with a single entry point and consisted of four key components:

• Environment Scanner: This component continuously monitors objects added to the WebXR scene, dynamically attaching appropriate loggers to them.
• T object and DP object logger: These loggers capture granular 3D spatial data related to user interactions, specifically "click," "focus initiation," and "removal" events. Crucially, based on the attack scenarios, these loggers could differentiate between intentional and unintentional events, which is vital for identifying dark pattern effectiveness.
• Cursor Event Logger: This component specifically captures simultaneous interactions originating from a single cursor, allowing for detailed analysis of user input.
• Camera and Gaze Logger: This component estimates the user's position and the direction of their gaze within the 3D scene, providing insights into their attention and focus.

To extract meaningful quantitative insights from the collected logs, four interaction metrics were developed:

• Presence Metric: Quantifies the user's focus on the given task, helping understand how immersed they were.
• Safe Engagement Metric: Measures the impact of attack conditions on the user's ability to interact safely and as intended with the task.
• Malicious Attention Metric: Specifically designed to quantify unintended clicks, serving as a direct measure of the success of attacks relying on misdirection.
• Blind Spot Rendering Fraction: Introduced to measure the success of peripheral exploitation attacks by quantifying how often advertisements were rendered outside the user's immediate field of view to inflate impressions.

By integrating 14 different attacks across four diverse WebXR applications (reading, gaming, shopping, and travel), and embedding this comprehensive logging framework, the researchers created a powerful tool to empirically demonstrate and analyze the subtle yet effective nature of UI-based dark patterns in WebXR.

Demo / Proof of Concept

▶ Watch: Detailed example: Malvertising attack with transparency (8:15)

While the talk did not feature a live, real-time demonstration of a single attack in isolation, the entire user study framework and the "suit of applications" developed for it served as a comprehensive proof of concept for the feasibility and effectiveness of the proposed UI attacks in WebXR. The researchers meticulously integrated all 14 identified attacks (including the 5 novel ones) into four distinct types of WebXR applications: reading, gaming, shopping, and travel. These applications were designed to simulate realistic WebXR environments with varying contexts and interaction demands.

For instance, the gaming app likely featured scenarios where Visual Overlapping or Malvertising could be employed, such as a hidden ad behind a game target or a transparent ad overlaying a clickable item. The shopping app might have showcased Functionality Disruption attacks, preventing users from adding items to a cart, or UI-based Privacy Leakage scenarios where deceptive UI elements tricked users into entering sensitive information. By having 100 participants interact with these instrumented applications under specific attack conditions (or control conditions), the study effectively demonstrated that these theoretical attacks could be practically implemented within existing WebXR frameworks (A-frame, 3JS) and, more importantly, that they were highly successful in achieving their malicious objectives, often without the user's conscious awareness. The data collected through the sophisticated logging framework and analyzed using the custom interaction metrics provided empirical evidence that these "shadowed realities" are not just theoretical constructs but tangible threats within the WebXR ecosystem.

Defensive Implications

▶ Watch: Investigating attack impact through user study (9:30)

The findings from "Shadowed Realities" provide critical insights for both WebXR developers and platform owners to enhance the security posture of immersive web experiences. The primary defensive implication is the urgent need for awareness and proactive design. Developers must recognize that WebXR's unique properties, while enabling rich interactions, also introduce new attack surfaces not present in the traditional web.

Here are key defensive strategies:

1. Rethink UI Design Principles with Security in Mind: Developers should be educated on the 14 identified security-sensitive UI properties (e.g., overlapping, transparency, synthetic input) and understand how they can be manipulated. Default design patterns should prioritize security over unchecked immersion. For instance, carefully consider the layering of interactive objects and the use of transparent elements, especially when third-party content like advertisements is involved.
2. Implement WebXR-Native Isolation Mechanisms: Platform owners (e.g., browser developers like Google, Microsoft, Meta) need to explore and develop WebXR equivalents of iframes or the same-origin policy. This could involve introducing secure containers for third-party content within a 3D scene, ensuring that scripts and interactions from different origins are properly isolated and cannot interfere with each other's UI elements or event handlers.
3. Stricter Content Policies and Ad Review: For WebXR advertising ecosystems, platform owners and ad service providers should implement more rigorous content review processes. This includes not only scanning for malicious code but also actively analyzing UI elements for potential dark patterns that exploit overlapping, transparency, or synthetic input to mislead users. Ads that rely on such deceptive practices should be rejected.
4. Enhanced User Feedback and Transparency: WebXR environments could integrate clearer visual cues or haptic feedback to indicate when an interaction is being redirected or when a transparent object is intercepting a click. For example, a subtle outline around the actual clickable element when a user's gaze hovers over it, or a notification when an auxiliary browser window opens due to an interaction within the 3D scene.
5. Develop Security-Aware Development Frameworks: Libraries like A-Frame and 3JS could integrate security checks or offer safer defaults that mitigate common UI attack vectors. This might involve warnings when developers create highly overlapping interactive elements or providing APIs that make it harder to accidentally create unc clickable bait objects that funnel clicks to hidden elements.
6. User Education: While developers and platforms bear the primary responsibility, educating WebXR users about the potential for UI-based deception in immersive environments can also play a role. However, as the study showed, many attacks go unnoticed, highlighting the limitations of relying solely on user vigilance.

By addressing these points, the WebXR community can work towards building a more secure and trustworthy immersive internet, ensuring that beneficial UI properties enhance user experience without inadvertently creating avenues for malicious exploitation.

Key Takeaways

• WebXR's unique architecture, particularly the absence of iframes and the same-origin policy, creates novel security vulnerabilities, especially in its advertising ecosystem.
• The research identified 14 security-sensitive UI properties (e.g., overlapping objects, transparency, synthetic input) that can be exploited to create dark patterns.
• A four-category taxonomy (Click Manipulation, Peripheral Exploitation, Functionality Disruption, UI-based Privacy Leakage) was developed, encompassing 14 UI-based attacks, including five novel ones like Visual Overlapping and Malvertising.
• A comprehensive user study with 100 participants demonstrated that most of these UI attacks go unnoticed by users but are highly effective in achieving malicious objectives, such as generating unintended clicks or inflating ad impressions.
• WebXR developers and platform owners must prioritize security by redesigning UI principles, implementing WebXR-native isolation mechanisms, enforcing stricter content policies, and enhancing user feedback to mitigate these pervasive threats.
• The study underscores the urgent need for the WebXR community to develop robust security frameworks to ensure a safe and trustworthy immersive internet experience.

About the Speaker(s)

Chandrika Mukherjee is a researcher from Purdue University. Her work, presented in this talk, "Shadowed Realities: An Investigation of UI Attacks in WebXR," was conducted in collaboration with researchers from American University of Sharjah and University of California Irvine. Her research focuses on understanding and addressing security challenges within extended reality (XR) environments, particularly in the context of WebXR and its potential for UI-based attacks and dark patterns.