Foreign Information Manipulation and Interference (Disinformation 2.0)

Foreign Information Manipulation and Interference (Disinformation 2.0)

Speakers: Frankie Sageran, Head of Digital Insights, Information Environment Analysis Team, NATO

Conference: Black Hat Asia

YouTube: https://www.youtube.com/watch?v=JjtF7gbguC4

Overview

In an era where information is both abundant and weaponized, Frankie Sageran, a seasoned expert from NATO's Information Environment Analysis team, delivered a critical talk at Black Hat Asia titled "Foreign Information Manipulation and Interference (Disinformation 2.0)." Sageran, with over 30 years of experience at NATO, including roles as Head of Social Media and Head of Digital Insights, provided a sobering look at the evolving landscape of state-sponsored disinformation campaigns, with a particular focus on the Asia-Pacific region. The presentation meticulously outlined the sophisticated methodologies employed by state actors to deceive, manipulate, and ultimately destabilize nations.

The talk underscored the profound gravity of foreign information manipulation and interference (FEMI), a threat that the World Economic Forum has identified as the top global risk for the coming two years, surpassing even climate events and armed conflicts. Sageran emphasized that modern disinformation extends far beyond simple falsehoods, representing a deliberate, highly coordinated effort by state-level adversaries to exploit societal vulnerabilities, influence political processes, and erode trust. By detailing historical precedents, contemporary tactics, and future challenges posed by artificial intelligence, the presentation served as an urgent call to action for cybersecurity professionals, policymakers, and the public alike to understand and counter this pervasive threat.

Background

▶ Watch: Defining disinformation, misinformation, and FEMI (0:40)

To properly contextualize the threat, Sageran began by defining disinformation as "deliberate, distorted information that is secretly leaked into the communication process in order to deceive and manipulate." This definition, he noted, originated from a defected KGB officer in the 1980s, highlighting the long-standing nature of such operations. He cited Operation Infection, a KGB campaign from the 1980s that falsely claimed the AIDS virus was leaked by the Pentagon, as an early example, demonstrating that sophisticated manipulation predates the internet and social media. Historically, KGB officers were required to dedicate 25% of their time to producing disinformation stories, illustrating the sheer scale of these efforts even decades ago. It is crucial to distinguish disinformation from mere misinformation, which involves the ignorant spreading of false information, or simply disagreeing with uncomfortable truths.

Sageran then introduced the concept of Foreign Information Manipulation and Interference (FEMI), which refers to non-illegal activities by foreign states to interfere in the processes of other countries. These efforts are characterized by manipulative, intentional, and coordinated patterns of behavior and content that threaten or impact political processes and values. While all FEMI may involve disinformation, not all disinformation constitutes FEMI, indicating a nuanced relationship between the two. The European External Action Service has attempted to visualize these networks, identifying Russian actors typically in red and Chinese actors in blue, with larger nodes indicating a higher frequency of involvement. A critical aspect of FEMI is the distinction between official accounts (e.g., Ministries of Foreign Affairs) and non-attributable accounts, which are fake personas or state-linked/aligned channels designed to obscure state sponsorship, making attribution challenging.

The methodology for detecting and analyzing disinformation, Sageran explained, often follows the ABCD model developed by James Spam from Lunt University in Sweden, recently expanded to ABCDE. This framework involves:

• Actors: Identifying who is behind the manipulation—nation states, commercial entities, or political groups.
• Behavior: Analyzing their online actions, such as creating fake personas, coordinated posting, or manipulating media (memes, videos).
• Content: Examining the actual narratives, which can include political messaging, supposedly hacked documents, scams, or fraud.
• Distribution: Determining where the content is spread, including social media platforms and fake websites.
• Effect: Attempting to measure the impact, which, while difficult to prove causally, can be correlated with shifts in public attitude or policy outcomes. This comprehensive approach underscores that successful disinformation campaigns require detailed intelligence on narratives, actors, threats, and their potential real-world impact.

Key Findings

▶ Watch: Understanding state actor networks and attribution (4:40)

Sageran highlighted several recurring patterns and sophisticated techniques employed by state actors in their disinformation campaigns. He noted that social media platforms, originally designed for connection, have become increasingly "murky waters" exploited by threat actors. With a growing number of people, including many Americans, now relying on platforms like TikTok for news, the impact of undetected disinformation becomes profound.

The core modus operandi of state-sponsored disinformation involves:

1. Exploiting existing cracks: Identifying and leveraging pre-existing societal divisions or vulnerabilities.
2. Creating a "big lie": Crafting an outrageous falsehood that is then "wrapped around a kernel of truth." An example cited was the false claim that Ukrainian President Zelensky's wife bought expensive jewels from Cartier in Paris; the kernel of truth was that she had indeed been in Paris.
3. Concealing their hand: Using fake personas or indirect channels to avoid official attribution.
4. Finding "useful idiots": Identifying unwitting individuals or groups who will amplify the message, believing it to be true.
5. Deny, Distract, Distort: The standard response if a campaign is exposed.
6. Playing the long game: These campaigns are not short-term efforts but are meticulously planned and executed over months or even years to slowly shift attitudes and perceptions, often well in advance of critical events like elections.

Sageran presented concrete examples of these techniques:

• Doppelganger websites: These are exact copies of legitimate news sites, such as The Guardian or even NATO's official website. The malicious sites feature subtly altered URLs (e.g., guardian.co.com instead of guardian.com). While many links on the page redirect to the genuine site, a specific section contains the fabricated story. These fake stories are then heavily promoted through paid advertisements on platforms like Facebook and Twitter, with one example involving over $100,000 in advertising spend, leading to more clicks than legitimate stories.
• Pink slime websites: These sites, such as the "Miami Chronicle" or "Boston Times," are designed to look like local news outlets but are entirely fabricated. Their purpose is to "whitewash" disinformation, giving it an appearance of local, credible news. China has adopted this playbook with initiatives like Paper Wall and Glass Bridge, creating fake news sites like the "Eiffel Post" and "Civilia Times" in Spain. Many of these domains were registered as early as 2019, targeting a global audience across Japan, Korea, the UK, and Latin America.
• Fake fact-checking organizations: Adversaries create their own "fact-checking" websites, such as Russia's waronfakes.com, which ironically serve to spread disinformation under the guise of debunking. This further complicates efforts to discern truth from falsehood.
• Operation Overload: This tactic involves bombarding genuine fact-checking organizations with hundreds of fake stories—ranging from fabricated graffiti in France to false claims about the war in Ukraine—to overwhelm their resources and prevent them from focusing on other critical issues.
• AI Model Infiltration: A particularly alarming finding was the Pravda network's successful infiltration of AI chatbots and large language models (LLMs). This operation involved publishing false claims across 150 domains, covering 27 different narratives, impacting nearly 50 countries, and pushing approximately 3.6 million articles annually into the information ecosystem. These efforts were distributed across four major social media platforms and translated into over 46 languages, effectively distorting how LLMs process news and information. The consequence is that Western AI systems, including tools like ChatGPT, may now incorporate this injected disinformation into their responses.

Technical Deep Dive

▶ Watch: The ABCD Model for detecting disinformation campaigns (6:00)

The technical sophistication of modern disinformation campaigns is rapidly advancing, moving beyond simple website defacements to intricate, multi-layered operations that exploit both human psychology and technological vulnerabilities.

Doppelganger Websites represent a prime example of this technical prowess. These operations involve:

• Domain Squatting and Typosquatting: Registering domain names that are either very similar to legitimate news outlets (e.g., guardian.co.com instead of guardian.com) or leveraging common typographical errors.
• Content Mirroring and Injection: Adversaries mirror the entire aesthetic and structure of target websites, including logos, fonts, and layout. Crucially, while most navigational links (what Sageran referred to as the "orange parts") correctly direct users to the legitimate site, specific, strategically placed sections (the "blue parts") contain the fabricated articles. This blend of authentic and fabricated content makes detection difficult for casual users.
• Targeted Advertising: The distribution of these doppelganger articles is not left to organic reach. Instead, significant financial investment is made in paid advertising campaigns on major social media platforms like Facebook and Twitter. Sageran noted one instance where over $100,000 was spent promoting a single fake story, demonstrating a willingness to invest substantial resources for broad dissemination.

The phenomenon of Pink Slime Websites and their Chinese counterparts, Paper Wall and Glass Bridge, illustrates a strategy of creating an entire ecosystem of fake news sources. These are not merely individual false articles but entire domains designed to mimic local or specialized news outlets. The technical aspect here involves:

• Mass Domain Registration: Adversaries register numerous domains with names that sound credible (e.g., "Miami Chronicle," "Eiffel Post"). Sageran noted that many Chinese domains were registered as early as 2019, indicating a long-term strategic investment.
• Generic Website Templates: These sites often use common content management systems or templates, making them quick and inexpensive to deploy at scale. The goal is quantity and perceived legitimacy rather than unique design.
• Content Laundering: By publishing disinformation on these seemingly independent "news" sites, state actors can then cite them as sources, giving the false narratives a veneer of journalistic credibility.

The integration of Artificial Intelligence (AI) represents the most significant technical advancement in disinformation. AI tools are now leveraged across multiple stages of a campaign:

• Automated Content Generation: AI can generate countless versions of a single narrative, tailoring it for different audiences and platforms. This dramatically increases the volume and velocity of disinformation dissemination.
• Synthetic Media (Deepfakes): AI is used to create increasingly realistic fake audio, images, and videos. Sageran emphasized that the quality of deepfakes has improved dramatically in the last year, making them "harder and harder every day to detect." He provided a hypothetical example of a deepfake video showing a manipulated interaction between political figures, highlighting the potential to completely alter perceptions of real-world events.
• AI Social Media Bio Generators and Automated Comments: AI can create believable fake social media profiles, complete with generated bios and profile pictures, and then deploy automated comments to amplify narratives, making bot activity harder to distinguish from genuine engagement.
• Large Language Model (LLM) Infiltration: The Pravda network example demonstrates a critical new vector: the deliberate poisoning of AI training data or influencing the information LLMs access. By publishing 3.6 million articles annually across 150 domains in 46 languages, adversaries are effectively injecting vast amounts of fake content into the global information stream. As LLMs continuously scrape and process this content to learn and generate responses, they inadvertently incorporate and propagate this disinformation, leading to Western AI systems potentially outputting state-sponsored narratives. This represents a profound shift, as the very tools we rely on for information processing can become unwitting conduits for manipulation.

Demo / Proof of Concept

▶ Watch: Social media's critical role in spreading disinformation (8:20)

While the presentation did not feature a live, interactive demonstration in the traditional sense, Sageran provided a compelling series of real-world case studies and examples that effectively served as proofs of concept for the discussed disinformation tactics. These examples illustrated the practical application and impact of sophisticated manipulation.

One notable example involved a NATO video about the war in Ukraine. State actors photoshopped an SS insignia onto a soldier's uniform in the video. This manipulated image was then disseminated in multiple languages (Russian, Spanish, Dutch, German) as a coordinated effort. The campaign gained further traction when the Russian embassy in South Africa amplified the false claim, lending it an air of official endorsement and suggesting that widespread sharing validated its authenticity. This demonstrated the power of coordinated cross-platform and cross-lingual dissemination, combined with official state backing, to legitimize fabricated content.

Another powerful case study detailed a multi-month campaign designed to block funding for Ukraine. This narrative began with false claims about the misuse of US funding, evolving through stories like "Zelensky's wife bought jewels from Cartier" and "Zelensky bought mansions in Florida." Sageran showed how the message successfully migrated from initial Russia supporters (represented by pink nodes in their analysis) to right-wing groups (green nodes), who then amplified the message. The campaign ultimately achieved its objective in December 2023 when Senate Republicans blocked a move to pass support for Ukraine, illustrating how sustained disinformation can directly influence legislative outcomes.

Closer to the Asia-Pacific context, Sageran presented two regional narratives:

• "South Korea is a failed US vassal state": Network analysis revealed significant bot activity supporting key nodes in this narrative. The actors involved were a combination of Russian and Chinese state supporters, demonstrating a coordinated effort between these nations to push specific geopolitical narratives.
• "The world must remember World War II era atrocities committed by Japan": This campaign also exhibited high bot activity, with close to 26% of all accounts involved being bots. The primary actors behind this narrative were identified as Chinese accounts, indicating an orchestrated attempt to amplify historical grievances for strategic purposes.

Finally, Sageran shifted to the realm of hybrid threats or hybrid warfare, where digital disinformation seamlessly transitions into real-world physical actions. He presented several alarming examples:

• French Coffins under the Eiffel Tower: After France discussed sending troops to Ukraine, five coffins draped with French flags appeared under the Eiffel Tower. The individuals responsible were recruited through Telegram channels and paid $100 in Bitcoin to carry out this act, demonstrating how digital recruitment can lead to physical acts of protest or intimidation.
• Anti-NATO Sticker Campaign in Belgium: A journalist, posing as an anti-NATO activist, was recruited via digital channels to place anti-NATO stickers around the European Parliament and was paid $40-50 in Bitcoin for photographic proof.
• Sabotage and Cyberattacks: Sageran cited widespread election interference (Bosnia, Moldova, Georgia), cyberattacks on hospitals, warehouse fires in Poland (including pharmaceutical companies), GPS jamming in Estonia (leading to plane diversions), a plot to place a firebomb on a DHL plane in Germany, and even plots to assassinate European weapons chiefs. He also mentioned Russian attempts to destabilize countries through seemingly innocuous acts like climate activists spraying car exhaust pipes.

These examples underscore that disinformation is not an isolated digital phenomenon but an integral component of a broader strategy to undermine, destabilize, and coerce nations, often culminating in tangible, real-world consequences.

Defensive Implications

▶ Watch: Common recurring patterns in state-sponsored disinformation (9:15)

Understanding the sophisticated nature of Foreign Information Manipulation and Interference necessitates a multi-faceted defensive strategy that spans individual behavior, educational initiatives, and international policy. Sageran outlined several critical implications for defenders:

First and foremost is the imperative to increase public awareness. Individuals must be educated about the existence of state actors behind disinformation campaigns, recognizing that these efforts are purposeful, designed to mislead, and aim to impact democratic processes. This awareness is particularly crucial around elections, although Sageran emphasized that such campaigns are continuous, not merely episodic.

Secondly, a significant shift in individual behavior is required. In an age of instant information sharing, people often prioritize being the first to post a story. Sageran urged a change: "Take some time to check is this a real story? Check the news outlets and don't just check social media channels." He advised cross-referencing information with reputable journalists and established news sources before sharing. This emphasis on critical thinking and verification at the individual level is a fundamental line of defense. The example of Finland, where children are taught in school how to detect disinformation, was highlighted as a proactive educational model that other governments should consider adopting into their curricula.

Finally, a more robust and coordinated counter-disinformation strategy is essential, moving beyond mere debunking. While debunking and pre-bunking (pre-emptively exposing disinformation tactics) are valuable, they are only part of the solution. Sageran advocated for:

• Enhanced International Cooperation: Countering state-sponsored disinformation requires a coordinated international effort, transcending individual country initiatives.
• Improved Domain Name Regulation: The ease with which fake domains can be registered allows adversaries to "whitewash" their information. Stricter regulations on domain name registration are needed to make it harder for malicious actors to establish seemingly legitimate online presences.
• Platform Accountability: Social media platforms must be held more accountable for the content disseminated on their services. Sageran noted their reluctance, often citing their role as mere service providers and the potential loss of advertising revenue (e.g., $100,000 for a single fake ad campaign). However, their role in amplifying these messages demands greater responsibility.
• Improved Data Access for Researchers: Currently, researchers often have limited access to data from major platforms, with Twitter (now X) being one of the few with relatively open firehose access. Sageran argued that if platforms like Meta were to open up their data to researchers for non-commercial purposes, the true scale and impact of the disinformation problem would likely be revealed to be "even bigger than what we actually think for the moment." This data access is vital for comprehensive threat intelligence and effective counter-strategies.

These defensive implications underscore that combating disinformation requires a holistic approach, integrating public education, individual vigilance, regulatory reforms, and collaborative efforts across technology platforms and international bodies.

Key Takeaways

• Disinformation is a Deliberate State-Sponsored Threat: It is not accidental but a highly organized, intentional effort by state actors to deceive, manipulate, and undermine democratic processes and societal stability, as highlighted by the World Economic Forum ranking it as a top global risk.
• Sophisticated Tactics Exploit Trust and Technology: Adversaries employ advanced methods like Doppelganger websites (cloned legitimate sites with fake content), Pink Slime websites (fabricated news outlets), and fake fact-checking sites to lend credibility to false narratives and confuse the public.
• The "Long Game" Exploits Cracks and Human Psychology: Disinformation campaigns are sustained over months or years, exploiting existing societal divisions, wrapping "big lies" around "kernels of truth," and leveraging "useful idiots" for amplification, making them difficult to detect and counter in the short term.
• AI Accelerates and Amplifies Manipulation: Artificial intelligence is rapidly evolving into a potent tool for disinformation, enabling mass content generation, increasingly undetectable deepfakes, automated social media personas, and critically, the infiltration and poisoning of Large Language Models (LLMs), potentially turning them into unwitting conduits for state-sponsored propaganda.
• Digital Disinformation Fuels Hybrid Threats: The line between online manipulation and real-world impact is blurring, with disinformation campaigns now integrated into hybrid warfare strategies that lead to physical sabotage, cyberattacks on critical infrastructure, election interference, and even the recruitment of individuals for real-world disruptive acts.
• A Multi-Layered Defense is Essential: Effective countermeasures demand increased public awareness, individual behavioral changes (critical verification before sharing), robust international cooperation, stronger platform accountability for content, improved regulations for domain name registration, and greater data access for researchers to fully understand and combat the problem.

About the Speaker(s)

Frankie Sageran is a distinguished expert in information environment analysis, currently serving on the Information Environment Analysis team at NATO. With an impressive career spanning over 30 years at NATO, Sageran has held various influential positions. Prior to his current role, he served as the Head of Digital Insights and previously as the Head of Social Media, where he was responsible for the organization's output communications. His extensive experience includes conducting similar research on NATO itself, providing him with a profound understanding of the intricacies of information manipulation and its impact on international security.