Automated Large-Scale Analysis of Cookie Notice Compliance

Automated Large-Scale Analysis of Cookie Notice Compliance

Speakers: Ahmed Bouhoula

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=unji4eRnF9g

Overview

The internet browsing experience is frequently disrupted by ubiquitous cookie notices, a direct consequence of privacy regulations like the European Union's General Data Protection Regulation (GDPR) and ePrivacy Directive. These regulations mandate explicit, freely given, unambiguous, and specific consent for the use of non-essential cookies. However, the practical implementation of these requirements on a vast scale has proven challenging, leading to widespread non-compliance and the proliferation of dark patterns that subtly manipulate users into accepting tracking. This talk by Ahmed Bouhoula presents a groundbreaking automated methodology for large-scale analysis of cookie notice compliance, addressing the limitations of previous studies which were often manual, restricted to specific consent providers, or only examined the first layer of cookie interfaces.

Bouhoula's research introduces a sophisticated framework leveraging machine learning models to classify interactive elements, explore multi-layered cookie notices, and accurately categorize cookies based on their tracking potential. By crawling an extensive dataset of 97,000 websites, the study uncovers alarming rates of non-compliance, including the setting of tracking cookies without any notice, insufficient declaration of data collection purposes, and the outright disregard of user rejections. The findings highlight a critical disconnect between regulatory intent and real-world implementation, particularly revealing a "facade of compliance" where more popular websites, while appearing compliant, are more aggressive in data collection and user choice disregard.

This work is crucial for several reasons. Firstly, it provides the most comprehensive empirical evidence to date regarding cookie notice compliance across a diverse web landscape, mitigating the selection bias inherent in prior research. Secondly, its automated, generalizable methodology offers a scalable solution for continuous monitoring and enforcement, a critical step given the significant fines associated with GDPR violations. Finally, by collaborating with data protection authorities, this research moves beyond academic analysis to directly inform and support regulatory action, aiming to foster a more privacy-respecting online environment for users.

Background

▶ Watch: Introduction to cookie notice problem and regulations (0:00)

The proliferation of online tracking technologies, particularly through HTTP cookies, necessitated robust privacy regulations to protect user data. In the European Union, the ePrivacy Directive (often called the "Cookie Law") dictates that consent must be obtained before storing or accessing information on a user's device, with exceptions for strictly necessary cookies. The GDPR then elaborates on the definition of valid consent, requiring it to be freely given, specific, informed, and an unambiguous indication of the data subject's wishes by a clear affirmative action. This means users must have a genuine choice, understand what they are consenting to, and actively agree, rather than passively accepting through inaction or being coerced.

Despite these clear legal frameworks, many websites continue to employ practices that fall short of these standards. A common issue is the use of dark patterns, which are user interface designs that intentionally steer users towards making choices that benefit the website operator at the expense of user privacy. Examples include making "Accept All" buttons prominent while hiding or complicating "Reject All" options, or using confusing language to obscure the implications of consent choices. Prior empirical studies have extensively documented these issues, revealing widespread non-compliance and the prevalence of dark patterns.

However, existing research faced significant limitations. Many studies relied on manual analysis, which is inherently time-consuming and thus restricted to small sample sizes, limiting the generalizability of their findings. Others focused exclusively on websites that implemented specific Consent Management Providers (CMPs), such as those adhering to the IAB Europe's Transparency and Consent Framework (TCF). While these frameworks aim to standardize consent, restricting analysis to them introduces a selection bias, as such websites might not be representative of the broader web. Bouhoula highlights that some prior work, for instance, by Binger et al., covered only 0.6% of their initially considered websites due to these constraints. Furthermore, most studies only analyzed the first layer of cookie notices, failing to interact with "Settings" or "Manage Preferences" buttons that often reveal additional, potentially hidden, consent options or detailed information about data collection purposes. The challenge, therefore, was to develop a scalable, automated, and comprehensive method to evaluate compliance across the entire web, regardless of implementation specifics, and to delve into the deeper layers of consent interfaces.

Key Findings

▶ Watch: Our automated machine learning approach for compliance analysis (3:25)

The large-scale analysis conducted by Bouhoula and his team across 97,000 websites revealed a pervasive culture of non-compliance with privacy regulations, despite the potential for substantial GDPR fines. The findings underscore that users' privacy choices are frequently disregarded, and transparency is often lacking.

A significant discovery was the widespread absence of cookie notices where they are legally required. The study found that in approximately one-third of websites where tracking cookies (dubbed "AA cookies" for Analytics or Advertising) were detected, no cookie notice was present at all. This directly violates Article 5(3) of the ePrivacy Directive, which mandates consent for non-essential cookies.

Even when cookie notices were present, their content often fell short of legal requirements. Roughly one-quarter of cookie notices failed to properly declare the data collection purposes for which cookies were being used. This lack of specificity undermines the "informed" aspect of GDPR consent, leaving users unaware of how their data is being utilized.

Perhaps the most alarming findings pertain to the disregard of user choices. The research revealed that in approximately three-quarters of cases, websites set tracking cookies even before the user had a chance to interact with the cookie notice. This pre-emptive tracking completely bypasses the requirement for freely given, explicit consent. Furthermore, for users who did take the time to interact with the cookie notice and explicitly reject all cookies, their choices were often ignored. The study found that in roughly two-thirds of these instances, websites continued to set tracking cookies despite the user's explicit rejection. This represents a direct violation of the user's right to refuse consent.

The analysis also uncovered a "facade of compliance" based on website popularity. More popular websites tended to exhibit better visible compliance, such as including a cookie notice and a clear "Reject" button. However, these same popular websites were often "more aggressive when it came to data collection" and "more likely to ignore user choices" once those choices were made. This suggests that while large organizations may present a compliant front, their underlying data collection practices remain problematic.

Finally, the study rigorously demonstrated the selection bias inherent in prior research that restricted analysis to websites implementing specific consent frameworks (e.g., IAB TCF). By comparing results from their broad crawl with those from a subset of TCF-implementing websites, Bouhoula showed that such restricted analyses lead to "significantly different" conclusions, thereby underestimating the true extent of non-compliance across the wider internet. Overall, the research concluded that approximately three-quarters of the analyzed websites violate at least one legal requirement related to cookie consent.

Technical Deep Dive

▶ Watch: Decision tree logic for detecting compliance violations (6:00)

To overcome the limitations of prior research, Ahmed Bouhoula's team developed a sophisticated, automated methodology capable of large-scale, in-depth analysis of cookie notice compliance. This involved a multi-stage process leveraging machine learning models, intelligent web crawling, and a structured decision-making framework.

The first critical step was to enable meaningful interaction with diverse cookie notices, regardless of their visual design or underlying implementation. This required the ability to identify and classify interactive elements within the notice. For this, the researchers collected and annotated a dataset of 2,400 samples, each labeled with one of six categories: accept, reject, settings, save, close, or link. This dataset was then used to train a BERT (Bidirectional Encoder Representations from Transformers) model. BERT, a powerful transformer-based machine learning model pre-trained on a vast corpus of text, was fine-tuned for this specific classification task. The model achieved "good performance," allowing the crawler to accurately detect consent options like "Accept," "Reject," and crucially, "Settings" buttons.

Once interactive elements were classified, the crawler could intelligently interact with the cookie notice. A key innovation was the ability to explore subsequent layers of the notice. If a "Settings" button was detected, the crawler would click it to uncover potentially hidden options and preferences, simulating a user's deeper engagement with the consent interface. After identifying all available consent options (e.g., "Accept All," "Reject All," "Save Preferences"), the crawler would systematically browse the website after clicking on each detected option. This allowed for the extraction of the corresponding set of cookies that were set in response to each specific user choice.

The next technical challenge was to accurately classify these extracted cookies to determine their tracking potential. Building upon prior work by Binger et al. from the same research group, a gradient boosting model was employed for cookie classification. This model was initially designed to categorize cookies into four labels: essential, functional, analytics, or advertising. For the purpose of this study, the model was retrained to predict a binary label: whether a website uses cookies for analytics or advertising. These were collectively referred to as "AA cookies" (Analytics or Advertising cookies), as these are the types generally requiring explicit consent. A website was classified as using AA cookies if two or more such cookies were detected. This retrained model achieved a high precision of "almost 90%," indicating its reliability in identifying tracking cookies.

With the data from the crawl (presence/absence of notice, interactive elements, user choices, and corresponding cookies), the researchers developed a set of decision trees to detect six specific types of violations and two types of dark patterns. For example, a key violation, "ignored reject," was detected if the system found a cookie notice, identified a "Reject" button, and subsequently detected two or more AA cookies being set despite the "Reject" button being clicked. Other violations included the absence of a notice for AA cookies, improper declaration of data collection purposes, and pre-emptive setting of AA cookies before any user interaction. The decision trees provided a structured and auditable way to systematically assess compliance based on the collected evidence.

Finally, to ensure the reliability of their findings for potential enforcement actions, the methods were "tuned to have low false positive rates." A manual end-to-end evaluation was conducted on 500 random websites to validate the automated system's accuracy. This validation confirmed high precision values, albeit sometimes at the cost of lower recall, prioritizing the accuracy of reported violations to make the results suitable for collaboration with data protection authorities. The entire system was deployed to crawl the top 10,000 websites in 15 European countries, resulting in a dataset of 97,000 unique websites for analysis.

Demo / Proof of Concept

▶ Watch: Popularity bias: facade of compliance vs. aggressive tracking (8:00)

While the talk did not feature a live, interactive demonstration in the traditional sense, the entire research project serves as a large-scale, automated proof of concept for the feasibility and effectiveness of their proposed methodology. The "demo" of this work is the comprehensive analysis performed on 97,000 websites, which stands as a testament to the system's capabilities.

The core of the proof of concept involved deploying their developed crawler and machine learning pipeline across a vast segment of the web. This encompassed:

1. Automated interaction: The crawler successfully navigated and interacted with diverse cookie notices, identifying "accept," "reject," and "settings" buttons using the BERT model.
2. Multi-layer exploration: It demonstrated the ability to click through "settings" panels to uncover hidden options, simulating a user's deep engagement.
3. Cookie extraction and classification: The system accurately extracted cookies set after various user choices and classified them using the gradient boosting model to identify "AA cookies" (Analytics or Advertising cookies).
4. Violation detection: The decision trees successfully identified and categorized six types of violations and two dark patterns based on the collected data.

The scale of the crawl, covering the top 10,000 websites in 15 European countries (yielding 97,000 unique domains), unequivocally demonstrated the system's ability to operate at an unprecedented scale compared to prior manual or restricted studies. Furthermore, the commitment to "low false positive rates" and the subsequent manual end-to-end evaluation on 500 random websites served as a rigorous validation of the automated system's accuracy and reliability. This validation step is critical for transitioning academic findings into actionable intelligence for enforcement bodies, effectively proving that the system can reliably identify non-compliant websites. The results of this large-scale application, showing that roughly three-quarters of websites violate at least one legal requirement, are the ultimate proof of concept for the methodology's power and relevance.

Defensive Implications

▶ Watch: Collaboration with authorities for enforcement and future steps (9:00)

The findings presented by Ahmed Bouhoula paint a stark picture of widespread non-compliance, necessitating a fundamental shift in how websites handle cookie consent. For website operators and organizations, the defensive implications are clear and urgent: prioritize genuine compliance over superficial adherence.

Firstly, organizations must ensure complete transparency and proper declaration of data collection purposes. The finding that a quarter of cookie notices failed to properly declare purposes indicates a critical gap. Websites should clearly and specifically articulate what data is collected, why it's collected, and how it's used, moving beyond vague statements like "improve user experience." This includes a detailed breakdown of different cookie categories (essential, functional, analytics, advertising) and their specific functions.

Secondly, and perhaps most critically, user choices must be respected without exception. The revelation that two-thirds of websites ignored explicit rejections is a severe breach of trust and legal mandates. Websites must implement robust technical mechanisms to ensure that when a user rejects non-essential cookies, those cookies are genuinely not set. This requires careful auditing of third-party scripts and consent management platforms to verify that they are functioning as intended and are not circumventing user preferences. Developers should prioritize the "Reject All" option to be as prominent and easy to use as "Accept All," actively avoiding dark patterns that nudge users towards less private choices.

Thirdly, organizations should conduct regular, automated audits of their own cookie consent mechanisms. Given the dynamic nature of web development and third-party integrations, a one-time setup is insufficient. Tools similar to the one presented in this talk could be adapted for internal use to continuously monitor compliance, detect pre-emptive cookie setting, and verify that consent preferences are consistently honored across all layers of the cookie notice. This proactive approach can identify compliance gaps before they lead to regulatory scrutiny.

Finally, the study's collaboration with enforcement agencies like the French data protection authority (CNIL) signals an increased likelihood of regulatory action. Websites should view these findings not just as academic observations but as a precursor to more stringent enforcement. Investing in robust Consent Management Platforms (CMPs) that are proven to be GDPR-compliant and regularly updated is crucial. Furthermore, organizations should educate their legal, marketing, and technical teams on the nuances of GDPR and ePrivacy requirements to foster a culture of privacy-by-design, rather than treating compliance as an afterthought. Failing to do so risks significant financial penalties, reputational damage, and erosion of user trust, as exemplified by the multi-million Euro fines levied against tech giants like Google and Meta.

Key Takeaways

• Widespread Non-Compliance: Approximately 75% of websites violate at least one legal requirement regarding cookie consent, despite strict regulations like GDPR and the ePrivacy Directive.
• Disregard for User Choices: A significant number of websites (around two-thirds) set tracking cookies even after users explicitly reject them, directly undermining privacy regulations.
• Pre-emptive Tracking: Roughly three-quarters of websites set tracking cookies before users even have a chance to interact with the cookie notice, bypassing the requirement for explicit consent.
• Facade of Compliance: More popular websites often appear compliant with visible requirements but are more aggressive in data collection and more likely to ignore user choices once expressed.
• Automated, Scalable Analysis: The research introduces a novel, machine learning-driven methodology (using BERT and gradient boosting models) for large-scale, multi-layered analysis of cookie notice compliance, overcoming limitations of prior manual or restricted studies.
• Imminent Enforcement: The collaboration with data protection authorities signals a move towards stronger enforcement, making robust and genuine compliance more critical than ever for website operators.

About the Speaker(s)

Ahmed Bouhoula is the speaker for this talk, presenting research from his paper "Automated Large-Scale Analysis of Cookie Notice Compliance." While the transcript does not provide specific details about his title or company, the depth and technical nature of the work suggest he is a researcher or academic. His work focuses on developing automated methods to analyze complex web phenomena, particularly in the realm of privacy and security, leveraging machine learning techniques to address real-world challenges in regulatory compliance. The mention of prior work by Binger et al. conducted in "our group" indicates his involvement with a research team dedicated to privacy-related studies. His collaboration with the French data protection authority (CNIL) and the nonprofit organization NOYB highlights his commitment to translating academic research into practical tools for privacy enforcement.