SIGuard: Guarding Secure Inference with Post Data Privacy

SIGuard: Guarding Secure Inference with Post Data Privacy

Speakers: Xinqian Wang, PhD student, RM University

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=bK8TQv3JTJE

Overview

The proliferation of machine learning as a service (MLaaS) has revolutionized how intelligence is consumed, offering sophisticated prediction capabilities through cloud-hosted neural networks. While this paradigm provides immense value, particularly in sensitive domains like medical imaging, it introduces significant privacy challenges. Traditional MLaaS workflows involve users submitting data to a cloud provider for inference, receiving a prediction and often a confidence vector in return. This interaction, however, exposes both the user's sensitive data and the model owner's valuable intellectual property to potential privacy breaches.

Recent advancements in secure inference aim to mitigate these concerns by employing cryptographic techniques like secure multi-party computation (MPC). These systems allow users to submit encrypted data and model owners to upload encrypted models, with the cloud performing computations on the encrypted data without learning its contents. The user ultimately receives only the inference result, while neither party learns anything about the other's input. Despite these robust protections for input and model privacy, a critical vulnerability persists: the privacy of the output itself. The prediction results, even when derived from a secure inference protocol, can still encode sensitive information about the training dataset, making them susceptible to prediction API attacks, most notably Membership Inference Attacks (MIA).

This talk introduces SIGuard, a novel framework designed to fortify secure inference systems against these output-level privacy threats. SIGuard addresses the critical gap left by existing secure inference solutions, which, while protecting inputs, fail to safeguard information revealed through prediction outputs. By adapting and extending existing plaintext defenses against MIA, SIGuard provides a robust, provably secure, and efficient mechanism to perturb inference results in an encrypted domain, thereby mitigating MIA without compromising prediction accuracy or requiring costly model retraining. The work highlights the unique challenges of integrating such defenses into MPC-based systems, particularly concerning threat model changes and the efficient realization of complex algorithms, offering significant improvements in security and performance.

Background

▶ Watch: ML-as-a-Service: privacy challenges for users and models (0:00)

The landscape of MLaaS has seen rapid growth, with major cloud providers offering ready-made intelligence through APIs. In this model, users leverage pre-trained neural networks hosted by cloud providers to perform predictions on their data. While convenient, this setup inherently creates privacy dilemmas. Users are hesitant to expose sensitive data (e.g., medical records) to a third-party cloud. Concurrently, model owners view their sophisticated, expensive-to-train models as valuable intellectual property, fearing unauthorized replication or theft by the cloud provider. This dual concern highlights the need for robust privacy-preserving mechanisms.

Secure inference emerged as a promising solution to these challenges. By utilizing cryptographic techniques, primarily secure multi-party computation (MPC), secure inference protocols allow multiple parties to jointly compute a function over their private inputs without revealing those inputs to each other. In the context of MLaaS, this means the user's data and the model owner's model remain encrypted throughout the inference process, with only the final prediction revealed to the user. Many recent works in secure inference leverage MPC for its efficiency in handling the complex computations of neural networks.

However, the speaker points out a significant blind spot: secure inference, in its current form, cannot protect against information leakage from the prediction results themselves. This vulnerability stems from a class of attacks known as prediction API attacks, with Membership Inference Attacks (MIA) being a prominent example and the focus of this research. MIA aims to determine whether a specific data sample was part of the model's training dataset by observing its prediction output, often the confidence scores. Such attacks exploit the fact that models often behave differently on data they have seen during training versus unseen data. When a model is "corrupted" by an MIA attacker, it can be queried in a black-box manner, and the revealed predictions can be leveraged to infer membership. Crucially, even in secure inference systems where inputs are encrypted, the final prediction outputs are eventually revealed to the user, making these systems susceptible to MIA.

Defenses against MIA in the plaintext domain typically fall into two categories: training-time defenses and inference-time defenses. Training-time defenses involve specific training approaches that inherently reduce inference accuracy. In contrast, inference-time defenses perturb the inference outputs by injecting carefully crafted noise, aiming to preserve prediction accuracy. Among these, Mamgard is highlighted as a suitable starting point for SIGuard. Mamgard's key observation is that MIA classifiers are vulnerable to adversarial examples. It works by adding a carefully crafted noise vector to a confidence score vector, transforming it into an adversarial example that misleads the attacker's classifier. Mamgard is attractive because it preserves prediction accuracy, does not require costly retraining, and applies directly to inference, thus not disrupting existing MLaaS pipelines.

Despite its advantages, Mamgard cannot be directly adopted in secure inference due to several fundamental difficulties:

1. Data Domain: Mamgard operates on plaintext data, whereas secure inference computes on encrypted data using MPC.
2. Threat Model: Mamgard primarily considers MIA in the plaintext domain, where the MLaaS user is compromised. Secure inference, however, deals with an adversary compromising the cloud server. SIGuard further expands this to consider scenarios where the compromised server colludes with the user, creating a significantly different and more potent threat model.
3. Algorithmic Complexity: Mamgard employs a recursive algorithm involving complex computations, making its efficient realization within an MPC framework a non-trivial challenge.

These difficulties underscore the need for a tailored solution like SIGuard, which can adapt and extend Mamgard's principles to the unique environment of secure inference while addressing the inherent complexities of MPC and evolving threat models.

Key Findings

▶ Watch: Challenges adapting existing MIA defenses to secure inference (4:00)

The research behind SIGuard yielded several critical findings that informed its design and demonstrated its necessity:

Firstly, the team investigated the vulnerability of MPC-based secure inference to MIA, specifically focusing on how the approximate nature of non-linear functions in MPC (such as softmax approximation) might affect output privacy. MPC protocols often approximate complex non-linear functions for efficiency, leading to "perturbed predictions" compared to their plaintext counterparts. The core question was whether MIA adversaries could still exploit these perturbed predictions. Evaluating five classical MIA attacks across five datasets and using four popular softmax approximations in secure inference, the findings were stark: the risk of MIA against secure inference remains significant. In some cases, the risk was observed to be even higher than in plaintext scenarios, suggesting that the approximations introduced by MPC do not inherently provide MIA resistance and can sometimes exacerbate vulnerabilities.

Secondly, the researchers tackled the challenge of efficiently realizing a Mamgard-like defense within an MPC framework. This involved designing two core protocols:

• The Secure Noise Optimization Protocol: This protocol is responsible for finding an optimal noise vector that can be added to the secret-shared confidence vector to perturb it effectively against MIA. The optimization process needs to occur over encrypted data, which is a significant departure from Mamgard's plaintext operations.
• The Secure Noisy Validation Protocol: This protocol ensures that the crafted noise vector not only misleads the MIA attacker but also preserves the original inference accuracy. It validates that the perturbation does not significantly alter the model's primary prediction. The design had to carefully consider the impact of softmax approximation on defense performance, as different approximations could influence the effectiveness of the noise.

Thirdly, SIGuard addresses a crucial threat model change and a burdened attack surface in secure inference. In plaintext MIA, adversaries typically only have access to perturbed confidence scores. However, in a secure inference setting, especially when collusion occurs between a compromised cloud server and the user, MIA adversaries gain auxiliary knowledge about the defense mechanism itself. This includes details about SIGuard's functionality and hyperparameters, as well as the original secure inference protocol. This deeper insight allows adversaries to attempt to bypass the defense. A particular concern was Mamgard's reliance on recursive while loops for optimizing loss functions. Directly implementing these in MPC would reveal the loop termination condition (e.g., when a certain value B becomes zero) to the colluding adversary. This leakage of iteration numbers could be exploited, as different data samples might require different numbers of iterations to find an optimal noise vector. The research found that even if the condition itself is protected, the adversary could still infer the number of iterations due to observing functionality in a colluding scenario. For instance, in an example shown, if the number of iterations was greater or equal to 160, the data sample was highly likely to be a non-member. This difference in iteration distribution between member and non-member data, caused by varying optimization hardness for each sample, creates an exploitable side channel.

To mitigate this critical leakage, SIGuard incorporates a key refinement: replacing the recursive while loops with fixed-iteration for loops. This approach fixes the number of iterations, preventing the leakage of information about optimization hardness. This introduces a trade-off: fewer iterations lead to faster computation but potentially weaker defense, while more iterations increase computation burden but strengthen the defense. Through careful evaluation, the team demonstrated that for datasets like Cifar-10, setting iterations to 10 with a learning rate of 0.8 could achieve performance close to random guessing against MIA, significantly reducing the computational overhead compared to larger iteration counts (e.g., 300 iterations). This refinement resulted in a more than 50 times improvement in runtime and bandwidth compared to a basic direct implementation, effectively addressing the efficiency and privacy concerns arising from the changed threat model.

Technical Deep Dive

▶ Watch: Introducing SIGuard: output privacy for secure inference (5:10)

SIGuard's technical architecture is built upon the foundation of secure multi-party computation (MPC), specifically leveraging replicated secret sharing for its cryptographic primitives. The system operates within a model involving three non-colluding cloud servers, where a semi-honest adversary is assumed to compromise at most one of these servers. This adversary adheres to the protocol but attempts to learn as much as possible from observed information. Crucially, SIGuard also considers a more potent collusion threat model where the compromised server collaborates with the user, giving the adversary deeper insights into the defense mechanism.

The core input to SIGuard is the set of encrypted logits output from an existing secure inference protocol. Logits are the raw, unnormalized outputs of the neural network before the application of a softmax function to produce probability distributions (confidence vectors). SIGuard's primary function is to craft and inject carefully designed noise into these encrypted logits. This perturbation process happens entirely in the encrypted domain. After the noise is added, the perturbed logits are passed through a secure softmax function, which also operates on encrypted data. The resulting perturbed confidence vectors are designed to mislead MIA attackers while preserving the original prediction label.

The choice of replicated secret sharing is central to SIGuard's security. In this scheme, a secret value X is divided into three shares (x1, x2, x3). Each of the three parties (cloud servers) receives one share. The security property ensures that any single party, or even any two parties (two out of three shares), cannot reconstruct or learn the original value of X. Only when all three shares are brought together can the secret be revealed. This distributed trust model underpins the privacy guarantees of SIGuard against a single semi-honest adversary.

The defense mechanism itself is inspired by Mamgard, which identifies that MIA classifiers are susceptible to adversarial examples. Mamgard's strategy is to add a meticulously crafted noise vector to the confidence score vector, transforming it into an adversarial example that causes the MIA attacker's classifier to misclassify the membership status. Adapting this to the MPC context requires significant innovation. SIGuard achieves this through two novel MPC protocols:

1. Secure Noise Optimization Protocol: This protocol's objective is to compute the optimal noise vector δ that, when added to the secret-shared logit vector, minimizes a specific loss function designed to make the sample appear like a non-member to an MIA classifier, while simultaneously ensuring the primary prediction remains unchanged. This optimization process involves iterative computations (gradient descent-like steps) on secret-shared data. The challenge here is efficiently realizing operations like comparisons, exponentiations (for softmax), and recursive updates within MPC, which traditionally incur high communication and computational overheads. The protocol must handle these operations without revealing intermediate values to any single server.

1. Secure Noisy Validation Protocol: After an optimal noise vector δ is determined, this protocol validates its effectiveness. It checks two main criteria:

• Defense Efficacy: Does the perturbed confidence vector successfully mislead the MIA attack, making it difficult to infer membership?
• Accuracy Preservation: Does the perturbation δ maintain the original model's prediction accuracy? That is, does the class with the highest confidence score after perturbation still correspond to the original predicted class? This is crucial to ensure that protecting against MIA does not degrade the utility of the MLaaS.

A critical technical detail lies in the handling of the softmax approximation within MPC. The softmax function, which normalizes logits into probabilities, is inherently non-linear and computationally expensive in MPC. Various approximations exist (e.g., polynomial approximations, piecewise linear functions), and their choice impacts both the efficiency of the secure inference protocol and the effectiveness of the MIA defense. SIGuard's design carefully considers these approximations, ensuring that the noise optimization and validation protocols can operate effectively regardless of the chosen secure softmax approximation.

The most significant technical innovation to counter the collusion threat model is the replacement of Mamgard's recursive while loops with fixed-iteration for loops. Mamgard's while loops terminate based on a condition that indicates convergence of the noise optimization. In a colluding scenario, revealing this termination condition or the number of iterations provides information about the "optimization hardness" of a specific data sample. Member data samples might converge faster or slower than non-member samples, creating a detectable pattern. By replacing these with fixed-iteration for loops, SIGuard ensures that the number of iterations is constant and publicly known, thus preventing this side-channel leakage. While this might mean some samples are "over-optimized" and others "under-optimized" compared to an ideal adaptive process, the research demonstrates that an appropriate fixed number of iterations can still achieve strong defense performance. For instance, in the Cifar-10 dataset, selecting 10 iterations with a learning rate of 0.8 proved sufficient to reduce MIA performance to near-random guessing, offering a 50x improvement in runtime and bandwidth over an adaptive, but leaky, approach. This design choice represents a sophisticated trade-off between privacy, efficiency, and defense effectiveness in the face of a strong adversary.

Demo / Proof of Concept

▶ Watch: Mamguard's core idea: adversarial examples for MIA defense (8:00)

While the talk did not feature a live, interactive demonstration of SIGuard, the speaker presented a comprehensive evaluation that serves as a robust proof of concept for the framework's effectiveness and efficiency. The evaluation was structured around two primary research questions, providing empirical evidence for SIGuard's capabilities.

The first evaluation question focused on whether SIGuard could effectively mitigate the risk of Membership Inference Attacks (MIA) in a secure inference setting. To address this, the researchers conducted experiments using five classical MIA attacks against five diverse datasets. They measured the performance of these attacks using metrics such as balanced accuracy, true positive rate (TPR), and low positive rate (LPR). The results conclusively showed that SIGuard significantly reduced the success rate of MIA to levels approaching random guessing. This demonstrates that the carefully crafted noise, optimized and validated within the encrypted domain, successfully perturbs the confidence vectors to mislead MIA classifiers without revealing sensitive membership information. The findings highlight SIGuard's ability to provide stringent output privacy guarantees, even against sophisticated MIA adversaries operating on predictions derived from secure inference.

The second evaluation question addressed the practical feasibility of integrating SIGuard into existing secure inference pipelines by assessing its efficiency. This is a crucial aspect, as cryptographic defenses often introduce substantial computational and communication overheads. The evaluation focused on whether SIGuard could perform its defense mechanisms without introducing "dominant overhead." The speaker presented data demonstrating that SIGuard achieves significant performance improvements compared to basic implementations. Specifically, by replacing adaptive while loops with fixed-iteration for loops and carefully selecting the number of iterations (e.g., 10 iterations for Cifar-10), SIGuard achieved more than 50 times improvement in runtime and bandwidth. This substantial efficiency gain ensures that SIGuard is not only effective but also practical for real-world deployment, allowing it to be seamlessly integrated into existing secure inference systems without rendering them prohibitively slow or resource-intensive. The evaluation effectively served as a proof that SIGuard’s design choices, particularly the fixed-iteration loops and optimized MPC protocols, successfully balance security with performance, making it a viable solution for protecting output privacy in MLaaS.

Defensive Implications

▶ Watch: SIGuard's key findings and open research questions (8:50)

SIGuard offers critical defensive implications for organizations deploying or utilizing Machine Learning as a Service (MLaaS), especially those handling sensitive data where secure inference is already employed.

Firstly, the most direct implication is the enhancement of end-to-end privacy. While existing secure inference solutions protect input data and model intellectual property, SIGuard extends this protection to the inference outputs themselves. This means that even if an adversary gains access to the final prediction confidence scores, they cannot reliably infer whether a specific data point was part of the model's training set. This is particularly vital for applications in healthcare, finance, or any domain where the training data might contain personally identifiable information (PII) or other highly confidential records. Defenders should recognize that secure inference alone is insufficient for complete privacy and integrate output privacy solutions like SIGuard.

Secondly, SIGuard provides a robust defense against Membership Inference Attacks (MIA) without requiring costly changes to the model training process. This is a significant advantage, as retraining large-scale neural networks can be prohibitively expensive and time-consuming. Instead, SIGuard operates at the inference stage, perturbing the output logits in an encrypted manner. This allows organizations to deploy MIA defenses as an add-on to their existing secure inference pipelines, minimizing disruption and resource expenditure. Defenders should prioritize inference-time defenses that maintain model accuracy and avoid retraining, making SIGuard an ideal candidate.

Thirdly, the research highlights the importance of considering collusion in threat modeling. Traditional secure inference often assumes a semi-honest adversary compromising only a server or only a user. SIGuard explicitly addresses scenarios where a compromised server colludes with the user, which significantly broadens the attack surface. Defenders need to update their threat models to account for such sophisticated adversaries, as simple plaintext defenses or basic MPC implementations might leak critical information (e.g., optimization iteration counts). SIGuard's innovative use of fixed-iteration for loops instead of adaptive while loops is a direct response to this, demonstrating a crucial architectural pattern for mitigating side-channel leakages in colluding scenarios. Organizations should review their MPC protocol designs for similar potential information leakages.

Fourthly, SIGuard underscores the impact of approximations in MPC on security. The use of approximate non-linear functions (like softmax approximations) for efficiency in secure inference protocols can, counter-intuitively, sometimes increase the risk of MIA. This suggests that simply using MPC does not automatically confer output privacy. Defenders should be aware that architectural choices made for efficiency in MPC can have unintended security consequences and should rigorously evaluate the privacy implications of such approximations. SIGuard provides a framework that can operate effectively even with these approximations, ensuring defense efficacy.

Finally, the efficiency gains demonstrated by SIGuard (over 50 times improvement in runtime and bandwidth) are crucial for practical adoption. High overheads are a common barrier to deploying advanced cryptographic techniques. By optimizing its protocols and making intelligent design choices, SIGuard makes output privacy a practical reality. Defenders looking to implement such protections should seek out solutions that demonstrate strong performance characteristics, as efficiency is often a prerequisite for real-world deployment. The principles and optimizations employed in SIGuard could also inform the design of defenses against other attribute inference or model inversion attacks, though the speaker notes that further design work would be needed for direct application.

In summary, SIGuard provides a blueprint for how to robustly secure the output stage of secure inference, urging defenders to adopt a more comprehensive view of privacy that extends beyond inputs to encompass the sensitive information potentially embedded within prediction results.

Key Takeaways

• Secure inference alone is not enough for full privacy: While secure multi-party computation (MPC) protects user inputs and model intellectual property, it does not inherently safeguard output privacy from attacks like Membership Inference Attacks (MIA).
• MIA risk remains significant, potentially higher, in secure inference: The approximations used for non-linear functions (e.g., softmax) in MPC can sometimes exacerbate the vulnerability to MIA, making output privacy a critical, unaddressed concern.
• SIGuard offers a practical, inference-time defense: It adapts the principles of Mamgard to the encrypted domain, injecting carefully crafted noise into prediction outputs to mislead MIA attackers without requiring costly model retraining or compromising prediction accuracy.
• Novel MPC protocols address unique challenges: SIGuard introduces Secure Noise Optimization and Secure Noisy Validation protocols to perform noise generation and validation entirely on encrypted data, a key innovation for operating within MPC.
• Collusion threat model drives design innovation: SIGuard specifically mitigates information leakage in scenarios where a compromised cloud server colludes with the user, replacing adaptive while loops with fixed-iteration for loops to prevent side-channel attacks related to optimization hardness.
• Significant efficiency gains make it viable: Through careful design and optimization, SIGuard achieves more than 50 times improvement in runtime and bandwidth, demonstrating that robust output privacy can be integrated into secure inference pipelines without dominant overhead.

About the Speaker(s)

Xinqian Wang is a PhD student at RM University. His research focuses on enhancing the security and privacy of machine learning systems, particularly in the context of secure inference and defending against sophisticated privacy attacks. This work on SIGuard exemplifies his contributions to developing practical and provably secure solutions for real-world machine learning as a service applications.