LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulation

LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulation

Speakers: Joshua C. Zhao; Atul Sharma; Ahmed Roushdy Elkordy; Yahya H. Ezzeldin; Salman Avestimehr; Saurabh Bagchi

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=KHr7IjpUn8U

Overview

This article delves into LOKI, a groundbreaking data reconstruction attack designed to compromise the privacy of Federated Learning (FL) systems through sophisticated model manipulation. Presented by Joshua C. Zhao and his collaborators from Purdue University and the University of Southern California at IEEE S&P, LOKI demonstrates that even with robust privacy-enhancing mechanisms like Federated Averaging (FedAvg) and Secure Aggregation (SecAgg) in place, sensitive user data can still be extracted at an unprecedented scale. The research challenges the prevailing assumption that these defenses are sufficient to prevent data leakage in decentralized machine learning environments.

The talk introduces LOKI as a linear layer leakage attack, a class of exploits that leverages the gradients of fully connected layers to infer original training data. However, LOKI significantly advances the state-of-the-art by overcoming critical scalability limitations that plagued previous methods. Through novel techniques such as split scaling with convolutional layers and the convolutional scaling factor (CSF), LOKI achieves high-quality data reconstruction across hundreds of clients, leaking thousands of images where prior work struggled to recover even a handful.

The implications of LOKI are profound for the security of Federated Learning. By showing that large-scale data breaches are possible even under stringent privacy protocols, the research underscores an urgent need for re-evaluation of current FL defense strategies. It highlights that relying solely on architectural and cryptographic aggregation methods may be insufficient, advocating for the integration of stronger, certifiable privacy mechanisms like Differential Privacy (DP) or comprehensive model verification techniques to truly safeguard user data in real-world FL deployments.

Background

▶ Watch: Introduction to Federated Learning and privacy goals (0:00)

Federated Learning (FL), initially proposed by Google in 2016, revolutionized machine learning by enabling model training on decentralized user data while ostensibly preserving privacy. In the FL paradigm, data remains local on client devices (e.g., smartphones), which train a machine learning model with their private datasets. Only the model updates—typically gradients or model state—are transmitted back to a central server for aggregation. This decentralized approach was designed to circumvent the need for raw data collection by a central entity, thereby enhancing user privacy.

Despite this architectural design, research has shown that user data can still be leaked from these model updates. To mitigate such risks, several techniques have been employed:

• Federated Averaging (FedAvg): While not primarily designed as a defense mechanism, FedAvg inherently makes attacks more challenging. Clients perform multiple local training iterations before sending a final, aggregated model update to the server. This process obscures the intermediate steps of local training, increasing the unknowns for an attacker attempting to reconstruct individual data samples.
• Secure Aggregation (SecAgg): This is a cryptographic defense specifically aimed at preventing individual client data leakage. SecAgg ensures that the central server can only observe the final, aggregate model update. Individual client updates are cryptographically encrypted such that the server cannot derive any information from them until they are combined. After aggregation, the server receives a decrypted, unencrypted aggregate update. The challenge with SecAgg is its scale; in scenarios with hundreds of clients, a single aggregate update might represent contributions from thousands of images (e.g., 100 clients * batch size 64 = 6,400 images). Reconstructing individual images from such a large, aggregated update is exceptionally difficult for conventional attacks.

LOKI falls under the category of linear layer leakage attacks. These attacks exploit the properties of fully connected (FC) layers in neural networks. The fundamental idea is that if an image activates a specific neuron, the gradients associated with the weights and biases of that neuron can be manipulated to reconstruct the input image. For instance, by dividing the weight gradient by the bias gradient of an activated neuron, it's theoretically possible to reconstruct the image that triggered that activation. However, a significant challenge arises when multiple images activate the same neuron; this results in a "failed reconstruction," where the output is a blended or unintelligible combination of the contributing images.

Prior work, such as the "Robbing the Fed" attack, attempted to improve the efficiency of linear layer leakage. This method proposed configuring the weights of an FC layer to measure specific aspects of an image, like its average pixel brightness. By aligning the biases of the FC layer to follow a distribution (e.g., Gaussian) corresponding to these image properties, biases could act as cutoffs for neuron activation. An image would activate a set of neurons whose biases were below a certain threshold. For reconstruction, a subtraction operation was required. A critical prerequisite for this method was that the weight and bias gradients from all neurons activated by the same image needed to be identical. This was achieved by modifying parameters in a subsequent FC layer.

However, this property proved impossible to maintain within the FedAvg paradigm. As model parameters continuously change after local iterations, ensuring consistent weight and bias gradients across multiple activated neurons becomes unfeasible. To address this, prior methods introduced sparse activation, using a double-sided threshold activation function to ensure that each image activated only a single neuron, simplifying reconstruction to a direct division. This also required scaling weights and biases to fit the desired distribution.

A more fundamental problem emerged with scalability. To handle a large number of images from many clients, the number of neurons (or "bins") in the FC layer had to increase significantly. For example, with 64 images per batch and 100 clients, an aggregate update would involve 6,400 images. To ideally reconstruct individual images, one might need thousands of neurons. As the number of neurons increased, the distance between the biases of subsequent neurons became infinitesimally small. When weights and biases were scaled by these tiny distances, their magnitudes grew excessively large. This led to floating-point precision problems: the actual gradients contributed by individual images became minuscule relative to the colossal magnitudes of the parameters. Consequently, much of the crucial information embedded in these gradients was lost during reconstruction, severely degrading image quality and preventing real-world scalability. As demonstrated in prior work, reconstruction quality sharply declined as the number of clients increased from 10 to 50, even for the "best eight reconstructions," indicating a severe limitation for large-scale FL systems.

Key Findings

▶ Watch: Understanding Linear Layer Leakage attacks (2:15)

LOKI introduces several innovative solutions to overcome the inherent scalability and precision challenges of prior linear layer leakage attacks against Federated Learning, particularly in the presence of FedAvg and Secure Aggregation. The core findings and contributions are:

1. Convolutional Layer Assistance and Split Scaling: LOKI's central insight is the strategic use of convolutional layers to assist in linear layer leakage. It employs a technique called split scaling, where the number of convolutional kernels is scaled based on the number of clients. Crucially, this allows the subsequent fully connected (FC) layer to remain fixed in size, eliminating the need to scale its weights and biases to excessively large magnitudes. This directly addresses and resolves the floating-point precision problems that crippled prior attacks when dealing with a large number of clients.

1. Identity Mapping Sets: To enable convolutional layers to effectively pass information to a fixed-size FC layer while supporting multiple clients, LOKI utilizes identity mapping sets. For instance, with an RGB image, three kernels can form a set to push through the red, green, and blue channels, respectively, effectively replicating the input image at the convolutional layer's output. By having multiple such sets (e.g., 10 sets for 30 kernels), LOKI can customize separate convolutional layer parameters for each client, ensuring isolated and effective processing.

1. Adaptation for Aggregated Bias Gradients: Because the FC layer size is fixed and its biases are shared, the bias gradients are aggregated across all clients. This means individual bias gradients cannot be used for reconstruction as in prior single-client attacks. LOKI adapts by demonstrating that the bias gradient is not strictly required for reconstruction. A modified reconstruction equation is employed that primarily relies on the weight gradients, scaling the reconstructed image to a 0-1 range. While this might result in images appearing slightly brighter than the ground truth, the visual distinguishability remains very high.

1. The Convolutional Scaling Factor (CSF): A simple yet powerful innovation, the CSF scales the input image up before it reaches the FC layer. This is counter-intuitive to prior beliefs, which suggested smaller gradients were better for FedSGD-like attacks. LOKI's insight is that larger gradients relative to the parameters are highly desirable in a FedAvg attack. By scaling the image up and proportionally scaling down the FC layer weights, LOKI ensures that the gradients generated are significantly larger relative to the parameter magnitudes. This "pushes" activated neurons away from the distribution more effectively, preventing subsequent images from activating the same neuron and thereby increasing the leakage rate and reconstruction quality. The CSF also stabilizes the convolutional kernel parameters across local iterations, maintaining their effectiveness.

1. Unprecedented Scalability and Leakage Rate: LOKI demonstrates remarkable scalability, showing no diminishing returns in leakage rate as the number of clients increases. In experiments with 100 clients, Loki successfully reconstructed the majority of images with very high quality. Quantitatively, Loki achieved a leakage rate of 83% (5,290 leaked images out of 6,400) across all clients, vastly outperforming prior work like "Robbing the Fed with model inconsistency," which was limited to attacking a single client and achieved less than 1% (50 images out of 6,400).

1. Superior Reconstruction Quality and Efficiency: The CSF significantly improves reconstruction quality, as evidenced by higher SSIM (Structural Similarity Index Measure) and PSNR (Peak Signal-to-Noise Ratio) scores, and lower LPIPS (Learned Perceptual Image Patch Similarity) scores. Furthermore, Loki's efficiency increases with client batch size, nearing perfect bin-to-leaked-image efficiency, whereas prior work's efficiency decreased. Loki can also leverage multiple training rounds to strengthen the attack and introduces only about half the model size overhead of prior work without sacrificing leakage rate.

Technical Deep Dive

▶ Watch: Prior work: 'Robbing the Fed' attack mechanism (3:20)

Federated Learning (FL) operates on the principle that client data remains local. A central server dispatches a global model to numerous clients, each of which trains the model on its private dataset. Instead of sending raw data, clients transmit only the model updates (gradients or updated model parameters) back to the server. The server then aggregates these updates to refine the global model. This decentralized paradigm, while privacy-centric in design, has been shown to be vulnerable to data reconstruction attacks.

Two key mechanisms aim to bolster privacy in FL: Federated Averaging (FedAvg) and Secure Aggregation (SecAgg). FedAvg dictates that clients perform several local training iterations before sending their model updates. This process makes it difficult for an attacker to infer intermediate training steps or individual data points from the final aggregated update. SecAgg, on the other hand, employs cryptographic techniques to encrypt individual client updates, ensuring that the server can only see the combined, aggregate update, not the contributions of individual clients. The challenge for attackers is to reconstruct original data from an update that might represent thousands of images from hundreds of clients, all cryptographically combined.

Prior linear layer leakage attacks exploit the gradients of a neural network's fully connected (FC) layers. The core idea is that if an image activates a specific neuron, the gradients of that neuron's weights ($W$) and biases ($B$) contain information about the input. Specifically, the relationship $W_{grad} / B_{grad}$ can, under ideal conditions, reconstruct the input image. However, a major impediment to this approach is when multiple images activate the same neuron. In such cases, the resulting reconstruction is a blend of these images, rendering it unintelligible—a "failed reconstruction."

The "Robbing the Fed" attack attempted to address this by configuring FC layer weights to measure specific image properties (e.g., average pixel brightness) and setting biases to act as activation cutoffs, effectively creating "bins" for images. An image would activate a set of neurons. For reconstruction, a subtraction operation was introduced. A critical requirement was that the weight and bias gradients across all neurons activated by a single image had to be identical. This was achieved through careful parameter manipulation in a subsequent FC layer. However, this condition proved impossible to maintain in FedAvg, where model parameters constantly change during local iterations.

To circumvent this, prior work introduced sparse activation, employing a double-sided threshold activation function. Unlike a standard ReLU (Rectified Linear Unit), which has a single-sided threshold, this function ensures that an image activates only a single neuron. This simplifies the reconstruction equation to a direct division. To maintain the distribution-measuring property, weights and biases were scaled.

The fundamental scalability bottleneck in prior methods arose from the need to increase the number of FC neurons (bins) to accommodate a growing number of clients and images, especially in SecAgg scenarios (e.g., 6,400 images from 100 clients, each with a batch size of 64). As the number of neurons increased, the distance between the biases of adjacent neurons became minuscule. To maintain the desired distribution, the weights and biases of the FC layer had to be scaled by these tiny bias differences, leading to their magnitudes becoming astronomically large. When small gradients, derived from individual images, were added to these colossal parameters, floating-point precision problems emerged. The significant difference in magnitude meant that much of the gradient information was lost or rounded off, severely degrading reconstruction quality and preventing any meaningful large-scale data recovery.

Loki's Innovation: Split Scaling and Convolutional Layers

LOKI fundamentally rethinks the architecture of linear layer leakage by leveraging convolutional layers and introducing split scaling. Instead of solely relying on a single, expanding FC layer, Loki uses convolutional layers to preprocess the input images before they reach a fixed-size FC layer. This is achieved through:

1. Identity Mapping Sets: For a typical three-channel RGB image, Loki employs sets of three convolutional kernels. Each kernel in a set is designed to pass through one color channel (e.g., one for red, one for green, one for blue), effectively creating an "identity mapping" that preserves the image information. If, for example, 30 kernels are used, this allows for 10 distinct identity mapping sets. This design is crucial because it allows Loki to customize the parameters of these convolutional kernels independently for each client, effectively isolating client contributions while maintaining a fixed FC layer size. This removes the problematic scaling of the FC layer weights and biases, directly addressing the floating-point precision issue.

1. Handling Aggregated Bias Gradients: With a fixed FC layer, its biases (and thus their gradients) are aggregated across all clients. This means individual bias gradients cannot be used for reconstruction. However, Loki demonstrates that the bias gradient is not strictly essential. The reconstruction can be performed primarily using the weight gradients. The modified reconstruction equation aims to scale the absolute maximum value of the weight gradient to 1, effectively mapping the reconstructed image to a 0-1 range. While this might result in images appearing visually brighter than their original counterparts (e.g., an image originally in 0-0.7 range scaled to 0-1), their content remains highly distinguishable.

The Convolutional Scaling Factor (CSF)

A seemingly simple but profoundly impactful aspect of Loki is the Convolutional Scaling Factor (CSF). Prior work often assumed that smaller gradients relative to parameters were beneficial for FedSGD-like attacks. Loki, however, operates on a critical insight: for FedAvg attacks, larger gradients relative to the parameters are highly desirable.

The CSF works by scaling the input image up by a certain factor before it enters the fixed FC layer. To maintain the overall scale of activations, the weights of the FC layer are proportionally scaled down. This manipulation ensures that the gradients generated by the FC layer are significantly larger in magnitude compared to the FC layer's weight parameters.

The benefits of a higher CSF are multi-fold:

• Higher Leakage Rate and Efficiency: When an image activates a neuron, the larger gradients (due to CSF) cause a more substantial shift in that neuron's parameters. This effectively "pushes" the activated neuron away from the distribution of other potential activations. Consequently, if a second image would have activated the same neuron, it is now less likely to do so. This mechanism prevents multiple images from activating the same "bin," leading to more successful single-image reconstructions and thus a higher overall leakage rate. This also means that Loki's efficiency—the number of leaked images per bin—actually increases with client batch size, approaching a near-perfect one-image-per-bin ratio.
• Enhanced Reconstruction Quality: The larger gradients induced by the CSF also stabilize the convolutional kernel parameters over multiple local iterations. If the CSF is small, the gradients can cause significant changes to the convolutional kernels, leading to distortions in the image propagated through them. With a large CSF, the kernels remain more consistent, ensuring that the image passed through them to the FC layer is a more faithful representation of the original, resulting in higher quality reconstructions.

Loki's architecture, combining split scaling with convolutional layers and the strategic use of the CSF, effectively bypasses the precision and scalability limitations of previous linear layer leakage attacks, enabling large-scale, high-quality data reconstruction even against advanced FL defenses.

Demo / Proof of Concept

▶ Watch: Loki's solution: Sparse Activation with double-sided threshold (5:00)

While the talk did not feature a live, interactive demonstration, the research thoroughly presented compelling experimental results that serve as a robust proof of concept for Loki's capabilities. These results unequivocally showcased Loki's superior performance in terms of reconstruction quality, scalability, and leakage efficiency compared to prior art.

Key demonstrations included:

• Visual Reconstruction Quality Comparison: The presentation included figures illustrating reconstructed images under challenging conditions, specifically with 100 clients participating in aggregation. Loki's reconstructions were consistently of very high visual quality, clearly identifiable, and retained significant detail. In stark contrast, prior work (even presenting its "best eight reconstructions") yielded images that were barely discernible, often blurry, and lacked critical details, demonstrating a near-total failure in large-scale scenarios. The speaker specifically noted that Loki reconstructed "the majority of images in all very high quality."
• Quantitative Metrics for Reconstruction Quality: The effectiveness of Loki and the impact of the Convolutional Scaling Factor (CSF) were quantified using standard image quality metrics. Figures presented showed that as the CSF value increased, Loki achieved:
• Higher SSIM (Structural Similarity Index Measure) scores, indicating greater perceptual similarity to the original images (higher SSIM is better).
• Lower LPIPS (Learned Perceptual Image Patch Similarity) scores, signifying less perceptual difference from the ground truth (lower LPIPS is better).
• Higher PSNR (Peak Signal-to-Noise Ratio) scores, another indicator of superior reconstruction quality (higher PSNR is better). These trends visually confirmed that the CSF not only enabled but actively enhanced the quality of reconstructed images.
• Scalability and Leakage Rate Comparison: Loki's most striking proof of concept was its ability to scale effectively. The talk highlighted a comparison:
• Prior work, specifically "Robbing the Fed with model inconsistency," was limited to attacking a single client and managed to leak only 50 images out of a total of 6,400 images contributed by all clients in an aggregated update (less than 1% leakage rate).
• Loki, on the other hand, was not restricted to a single client and successfully attacked all clients, achieving an astounding 5,290 leaked images out of 6,400, which translates to nearly an 83% leakage rate. This dramatic difference underscores Loki's unprecedented scalability.
• Efficiency with Client Batch Size: Graphs demonstrated that unlike prior methods, whose efficiency decreased as client batch size increased, Loki's efficiency improved. The number of images Loki could leak continued to increase with larger batch sizes, approaching "almost perfect bin to leaked image efficiency," meaning nearly one image per bin. This was attributed to the CSF effectively pushing neurons away from the distribution after activation.
• Overhead Comparison: The research also presented evidence that Loki achieved its superior leakage with significantly less overhead. It added "only about half the model size overhead of Prior work" without sacrificing any of the leakage rate, making it a more efficient and practical attack vector.

These experimental results, supported by visual comparisons and quantitative metrics, provided compelling evidence for Loki's efficacy and its capacity to perform large-scale data reconstruction in Federated Learning environments.

Defensive Implications

▶ Watch: The scaling problem with increasing neurons for clients (6:15)

The LOKI attack presents critical and sobering implications for the security and privacy guarantees of Federated Learning (FL). Its ability to reconstruct a large volume of high-quality data from aggregated model updates, even when protected by mechanisms like FedAvg and Secure Aggregation, fundamentally challenges the assumption that these techniques alone are sufficient to preserve user privacy.

The primary defensive implication is that Secure Aggregation, while a valuable cryptographic primitive, is insufficient as a standalone privacy-preserving mechanism in Federated Learning. The cryptographic encryption and aggregation of individual updates prevent the server from seeing raw client contributions, but LOKI demonstrates that information leakage can still occur after aggregation, from the final model state or aggregate gradients. This suggests that the aggregation process itself, despite its cryptographic protections, does not fully sanitize the updates of all sensitive information.

Defenders in FL systems must now recognize that:

1. Data Reconstruction is Scalable: The notion that large-scale data reconstruction is impractical due to the sheer volume of data and the complexity of aggregation is debunked by LOKI. Attacks can now realistically target and succeed in leaking thousands of images from hundreds of clients.
2. Architectural Defenses Have Limits: Relying solely on the decentralized architecture of FL and the obfuscation provided by local iterations in FedAvg is no longer adequate. More robust, mathematically provable privacy guarantees are required.
3. Malicious Clients are a Significant Threat: LOKI operates by manipulating the model structure and parameters at the client side. This highlights the severe threat posed by malicious or compromised clients who can intentionally craft updates to facilitate data extraction.

To effectively counter attacks like LOKI, the talk strongly advocates for the adoption of more robust and certifiable privacy-enhancing technologies (PETs):

• Differential Privacy (DP): The most emphasized defense is Differential Privacy. DP adds carefully calibrated noise to either the client's local data, gradients, or the aggregated model updates. This noise provides a mathematical guarantee that the presence or absence of any single individual's data in the training set will not significantly alter the outcome of the model, thereby preventing reconstruction attacks. DP offers a strong, certifiable privacy guarantee that goes beyond the architectural and cryptographic protections of FedAvg and Secure Aggregation.
• Verification of the Model: The talk also mentions "verification of the model" as a necessary defense. This could encompass several strategies:
• Anomaly Detection: Implementing server-side anomaly detection to identify unusual or suspicious model updates that might indicate malicious manipulation (e.g., unusually large gradients or parameter values as seen with Loki's CSF).
• Integrity Checks: Employing mechanisms to verify the integrity and structure of the model updates received from clients, ensuring they conform to expected norms and have not been tampered with in ways that facilitate leakage.
• Secure Multi-Party Computation (SMC) or Homomorphic Encryption (HE) beyond simple aggregation: While Secure Aggregation is a form of SMC, more advanced applications could allow for complex operations on encrypted data, potentially enabling checks or transformations that further obscure individual contributions without decrypting them at any single point.

In conclusion, LOKI serves as a critical wake-up call, demonstrating that the privacy promises of Federated Learning are not inherently fulfilled by current standard practices. A multi-layered defense strategy, with Differential Privacy at its core and supplemented by rigorous model verification techniques, is essential to truly achieve the privacy-preserving goals of FL and protect sensitive user data from sophisticated reconstruction attacks.

Key Takeaways

• LOKI is a powerful and scalable data reconstruction attack against Federated Learning, capable of extracting high-quality private data even with FedAvg and Secure Aggregation in place.
• The attack leverages convolutional layers and a technique called split scaling to overcome prior limitations related to floating-point precision and scalability, allowing it to attack hundreds of clients simultaneously.
• The Convolutional Scaling Factor (CSF) is a crucial innovation that significantly boosts Loki's leakage rate, improves reconstruction quality (higher SSIM/PSNR, lower LPIPS), and enhances efficiency by preventing multiple images from activating the same neuron.
• Loki achieves an unprecedented leakage rate of nearly 83% (5,290 out of 6,400 images) across all clients, far surpassing prior work which struggled to leak even 1% of images from a single client.
• The research unequivocally demonstrates that Secure Aggregation alone is insufficient to protect user privacy in Federated Learning, highlighting a fundamental vulnerability in current FL security paradigms.
• To truly achieve privacy-preserving Federated Learning, stronger, certifiable defenses like Differential Privacy are necessary, alongside potential strategies for model verification to detect and mitigate malicious client behavior.

About the Speaker(s)

The talk "LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulation" was presented by Joshua C. Zhao. This work was a collaborative effort involving researchers from Purdue University and the University of Southern California. The full list of speakers and collaborators on this paper includes: Joshua C. Zhao, Atul Sharma, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Salman Avestimehr, and Saurabh Bagchi. Their collective expertise spans the fields of machine learning, security, and privacy, particularly within the context of decentralized systems like Federated Learning.