The Inventory is Dark and Full of Misinformation: Understanding Ad Inventory Pooling in the Ad-Tech Supply Chain

The Inventory is Dark and Full of Misinformation: Understanding Ad Inventory Pooling in the Ad-Tech Supply Chain

Speakers: Yash Vekaria, Rishab Nithyanand, Zubair Shafiq

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=_yIlCzKRT4E

Overview

In the complex and often opaque world of online advertising, a deceptive practice known as dark pooling allows misinformation websites to clandestinely monetize their content, often at the unwitting expense of reputable brands. Presented by Yash Vekaria at IEEE S&P, this research, co-authored with Rishab Nithyanand and Zubair Shafiq, delves into the mechanics and prevalence of dark pooling, highlighting how it undermines industry transparency standards and renders conventional brand safety measures ineffective. The core issue revolves around ad networks bundling advertising inventory from diverse publishers, including both legitimate and illicit sites, making it nearly impossible for advertisers to discern where their ads ultimately appear.

The talk introduces dark pooling through a relatable analogy: imagine buying mangoes from a retailer, unaware that the crates contain a mix of good produce from farm A and spoiled produce from farm B, all bundled by an intermediary broker. Similarly, in the ad tech supply chain, advertisers purchase ad slots without full visibility into the true source of the inventory, inadvertently funding problematic websites. This research is the first comprehensive study to quantify the extent to which misinformation websites exploit dark pooling, revealing a significant gap in the industry's ability to ensure brand safety and combat the spread of harmful content. The findings underscore a critical need for enhanced compliance with existing transparency standards and the development of more robust mechanisms to safeguard advertising integrity.

Background

▶ Watch: Introduction to Dark Pooling and its deceptive monetization (0:00)

The online advertising ecosystem is a multi-layered supply chain connecting publishers who want to sell ad space with advertisers who want to buy it. At its heart are Supply-Side Platforms (SSPs), which help publishers sell their ad inventory, and Demand-Side Platforms (DSPs), which assist advertisers in buying it. These platforms converge at ad exchanges, where ad slots are auctioned in real-time. A common practice within this ecosystem is pooling, where ad networks or intermediaries aggregate ad inventory from multiple publisher websites. This can be a legitimate strategy, for instance, when child companies of a parent organization share a common seller ID to efficiently manage their collective ad inventory. This is classified as homogeneous pooling.

However, pooling becomes problematic when it involves dark pooling, a deceptive practice where publisher IDs or seller IDs are shared by organizationally unrelated entities to monetize ad inventory. In such scenarios, advertisers might believe they are purchasing ad space on a reputable website, but their ads are actually displayed on a misinformation site or another undesirable domain. The researchers classify pools as heterogeneous if there is no public documentation of a relationship between entities, and unauthorized if no such documentation can be found at all. Both heterogeneous and unauthorized pools are considered dark pools in this study, representing a significant transparency challenge.

To mitigate such transparency issues, the Interactive Advertising Bureau (IAB) introduced two key standards: ads.txt and sellers.json. ads.txt files, hosted at the root domain of a publisher website, are designed to list all authorized sellers of that publisher's inventory. Conversely, sellers.json files, hosted at the root domain of an SSP or ad network, are meant to list all the publishers whose inventory they are selling. Unfortunately, the talk highlights that the adoption and diligent compliance with these standards are poor, and even when present, they are plagued by misrepresentations that actively hinder transparency.

Advertisers typically rely on brand safety features to prevent their ads from appearing on problematic websites. These features often involve filtering or blocking specific publisher IDs, seller IDs, domains, or keywords. However, dark pooling presents a formidable challenge to these traditional brand safety mechanisms. If an advertiser blocks a seller ID associated with a dark pool, they risk blocking all publishers within that pool, including legitimate ones, leading to unacceptable "collateral damage." Similarly, domain or keyword-based filtering struggles when the true origin of the ad inventory is obscured by misrepresentation within the pooled environment. This inherent weakness in existing brand safety measures underscores why dark pooling is such an effective mechanism for misinformation sites to bypass detection and secure funding.

Key Findings

▶ Watch: Scenario: How dark pooling deceives advertisers (2:50)

The research aimed to answer three pivotal questions: assessing compliance with IAB transparency standards, quantifying the prevalence of dark pooling on misinformation websites and the broader web, and evaluating the effectiveness of brand safety services against this threat. The findings reveal a landscape rife with non-compliance and deceptive practices.

Firstly, regarding compliance with IAB's ads.txt and sellers.json standards, the study found significant misrepresentations, particularly on misinformation websites. ads.txt misrepresentations were observed to be far more common on misinformation sites compared to control (non-misinformation) websites. The disparity was even more pronounced for sellers.json files: an alarming 54.8% of sellers.json files that listed misinformation websites contained invalid domain names, and 46.1% included confidential entries. These misrepresentations severely impede a buyer's ability to reliably trace the source of ad inventory. The researchers also expressed disappointment that over half of the ad network sellers.json files they analyzed listed at least one misinformation website. Notably, the ad network Ref Content was found to have the highest number, listing over 200 misinformation sites in its sellers.json file.

Secondly, the study investigated the prevalence of dark pooling. Through a static analysis of ads.txt and sellers.json files across the top 100,000 websites, the researchers identified approximately 79,000 pools. A significant 11% of these were classified as dark pools, meaning they contained at least one misinformation website. The characteristics of these misinformation pools were distinct: they were found to be two times more likely to be heterogeneous (lacking a publicly documented organizational relationship) and remarkably 10 times larger in size than pools that did not include misinformation sites. The average heterogeneous misinformation pool encompassed around 500 websites. To further validate these static findings, a dynamic analysis was conducted by loading misinformation websites in a web browser, which confirmed the existence of approximately 300 unique misinformation pools. Ad networks such as 33 Across and Gour Ads were identified as owners of the most dark pools containing misinformation websites, with one pool by 33 Across featuring as many as 30 misinformation sites, and one by Gour Ads including 23. The ad exchange most frequently associated with these dark pools of misinformation websites exhibited a significant concentration of such activity, indicating a systemic issue.

Finally, the research analyzed the effectiveness of brand safety services by examining reputable brands whose ads appeared on misinformation websites. The team collected a total of 4,200 ads from approximately 2,000 distinct brands displayed on misinformation sites. A brand was classified as reputable if its domain ranked among the top 1,000 sites globally. The study observed a non-trivial number of reputable brands inadvertently advertising on these problematic websites. Crucially, the analysis revealed a higher conditional probability of an ad from a reputable brand ending up on a misinformation website if that site was part of one or more dark pools, compared to misinformation sites not participating in dark pooling. This means that reputable brands are disproportionately exposed to misinformation sites that leverage dark pooling. Disclosures made to these reputable brands confirmed that they were largely unaware their advertising dollars were buying dark-pooled inventory on misinformation sites and unequivocally did not wish to support such content.

Technical Deep Dive

▶ Watch: Why traditional brand safety fails against dark pooling (6:00)

The research employed a multi-faceted methodology combining both static and dynamic analysis techniques to comprehensively understand dark pooling. The initial phase focused on data collection for static analysis. This involved curating a robust list of known misinformation websites, cross-referenced with a control group of non-misinformation sites. For each website in both lists, the researchers programmatically crawled their respective ads.txt files. From these files, they extracted seller domains, categorizing them as either direct sellers, intermediaries, or both. Subsequently, the corresponding sellers.json files for these extracted seller domains (representing SSPs or ad networks) were crawled. This static data provided the foundational mapping of authorized sellers and the publishers they represented, allowing for the initial detection of pooling relationships.

A critical aspect of the technical approach was the precise classification of ad inventory pools. The researchers defined pools based on the organizational relationship between entities sharing a seller ID. A homogeneous pool was identified when entities shared the same parent owner, indicating a legitimate and transparent relationship. Conversely, a pool was classified as heterogeneous if public documentation of a relationship between the entities could not be found, suggesting a potential lack of transparency. If no public documentation could be found at all, the pool was labeled as unauthorized. Both heterogeneous and unauthorized pools were then designated as dark pools, representing instances where deceptive monetization could occur. This classification framework allowed the researchers to systematically distinguish between legitimate inventory sharing and potentially fraudulent activities.

To confirm the existence and active nature of these statically identified pools, the research proceeded with dynamic analysis. This involved loading the identified misinformation websites within a controlled web browser environment. During this process, all network traffic generated by the browser was meticulously collected. Filter lists, akin to those used by ad blockers, were employed to identify and isolate advertising traffic, often characterized by specific triplets of domain, seller ID, and publisher ID. A further step involved programmatically clicking on ads displayed on these web pages to record the landing pages of the click navigation. This dynamic observation provided empirical evidence of ads being served through dark pools and allowed the researchers to trace the actual user experience, verifying the ad placements on misinformation sites.

The study also detailed various types of misrepresentations found within the IAB transparency standards. While the full list is elaborated in their paper, the presentation highlighted common issues like the inclusion of invalid domain names and confidential entries within sellers.json files. These misrepresentations are not mere compliance oversights; they actively obscure the true identity of publishers and the relationships within ad networks, directly contributing to the opacity that dark pooling thrives upon. By quantifying the prevalence of these misrepresentations (e.g., 54.8% invalid domains, 46.1% confidential entries in sellers.json files listing misinformation sites), the research underscored the systemic nature of the transparency problem. The methodology for identifying reputable brands, based on being ranked among the top 1,000 sites, and the use of conditional probabilities to quantify the impact of dark pooling on reputable brand exposure, provided a robust statistical foundation for their key findings.

Demo / Proof of Concept

▶ Watch: Overview of the research questions addressed (7:10)

While the presentation did not feature a live, interactive demonstration of an exploit tool or a specific proof-of-concept application in the traditional sense, the entire research methodology served as a comprehensive proof of concept for detecting and quantifying dark pooling. The described static and dynamic analysis techniques effectively demonstrated that it is feasible to identify the opaque relationships within the ad tech supply chain, uncover non-compliance with IAB standards, and trace the flow of advertising dollars to misinformation websites. The systematic data collection, pool classification, and dynamic ad-serving verification steps collectively validated the existence and impact of dark pooling, substantiating the researchers' claims without the need for a separate, isolated demo.

Defensive Implications

▶ Watch: Finding: Prevalence of misinformation in ad networks (10:00)

The findings of this research carry significant implications for various stakeholders within the ad tech ecosystem, highlighting critical areas where defensive strategies must be re-evaluated and strengthened.

For advertisers and Demand-Side Platforms (DSPs), the primary takeaway is that current brand safety features are largely insufficient against the sophisticated obfuscation of dark pooling. Simple blocking of publisher or seller IDs can lead to unacceptable collateral damage, inadvertently blacklisting legitimate publishers. Advertisers must demand greater, more granular transparency from their SSPs and ad networks regarding the provenance of their ad inventory. This includes pushing for full disclosure of publisher relationships within pools and scrutinizing sellers.json files for misrepresentations. Implementing more sophisticated, AI-driven vetting processes that can analyze complex network relationships and flag suspicious pooling patterns is crucial to avoid unknowingly funding misinformation.

Supply-Side Platforms (SSPs) and ad networks bear a substantial responsibility in mitigating dark pooling. They must significantly improve their compliance with IAB standards like ads.txt and sellers.json, moving beyond superficial adoption to rigorous enforcement. This means actively preventing and correcting misrepresentations such as invalid domain names and confidential entries in their sellers.json files. SSPs need to implement stringent vetting processes for publishers they onboard into their networks, particularly those sharing seller IDs, to ensure legitimate organizational relationships. Developing internal mechanisms to detect and prevent unrelated entities from deceptively sharing seller IDs is paramount to restoring trust in the ad supply chain.

The Interactive Advertising Bureau (IAB) and the broader ad industry must acknowledge the systemic failures in existing transparency standards and their enforcement. There is a clear need for not only better compliance with ads.txt and sellers.json but also for the development of new or improved standards that are more resilient to obfuscation and misrepresentation. These new standards should focus on providing advertisers with unequivocal visibility into the ultimate destination of their ads. Furthermore, the industry needs to develop and promote effective disclosure and notification mechanisms that proactively inform brands when their ads are placed on problematic sites, empowering them to take immediate action.

Finally, the research underscores the growing role of regulatory intervention. Initiatives like the EU Digital Services Act (DSA), which is already in effect and imposes requirements on large online platforms to assess how their advertising systems are manipulated, set a precedent. The proposed US Digital Services Act (DZA) aims for similar transparency requirements. The researchers believe that regulatory mandates for enhanced ad transparency, specifically targeting ad networks and ad exchanges, can provide the necessary impetus for systemic change. Such regulations can empower brands, who are often the funders but lack control over ad placement, to demand greater accountability and ensure their advertising dollars align with their values.

Key Takeaways

• Widespread Non-Compliance: Misrepresentations in IAB ads.txt and sellers.json standards are rampant, particularly on misinformation websites, severely hindering transparency in the ad tech supply chain.
• Prevalence of Dark Pooling: Dark pooling is a significant issue, with approximately 11% of observed ad inventory pools containing at least one misinformation website. These misinformation pools are often heterogeneous and considerably larger than legitimate pools.
• Unwitting Brand Funding: Reputable brands, including many from the top 1,000 global sites, are inadvertently funding misinformation websites because their ads are placed through dark-pooled inventory, often without their knowledge or consent.
• Ineffective Brand Safety: Current brand safety measures, relying on blocking publisher or seller IDs, are largely ineffective against dark pooling due to the "collateral damage" of also blocking legitimate publishers within a shared pool.
• Need for Enhanced Standards & Compliance: There is a critical need for significantly improved compliance with existing transparency standards and the development of new, more robust standards to prevent ad fraud and provide granular visibility into ad placements.
• Regulatory Imperative: Regulatory frameworks, such as the EU DSA and proposed US DZA, are vital to enforce greater ad transparency from ad networks and exchanges, empowering advertisers and combating deceptive monetization practices.

About the Speaker(s)

Yash Vekaria, Rishab Nithyanand, and Zubair Shafiq are the researchers behind this impactful study on dark pooling in the ad-tech supply chain, presented at IEEE S&P. Yash Vekaria delivered the talk, outlining the methodology, findings, and implications of their work. While specific titles and affiliations beyond their association with this research were not detailed in the provided transcript, their collective effort highlights a significant contribution to understanding and addressing ad fraud and misinformation monetization in the digital advertising ecosystem.