How Federated Learning Protects Privacy, and Where It Stops

The honest answer to how federated learning protects privacy is: by architecture, and only partially. Federated learning trains a shared model across many devices or organizations without ever collecting the raw training data in one place. Each participant computes an update locally and sends only that update to a coordinating server. The data stays put. That is a real privacy property, and it is the reason regulators and standards bodies treat federated learning as a privacy-enhancing technology. It is not, on its own, a privacy guarantee, because the updates themselves can leak what the raw data contained. Understanding both halves of that sentence is the difference between deploying a genuine safeguard and deploying a compliance story.

What federated learning actually does

The canonical design comes from McMahan et al. (2016), the Google paper that coined the term. The motivating problem was mobile: phones hold text, photos, and usage data that is “privacy sensitive, large in quantity, or both, which may preclude logging to the data center.” Their answer, the Federated Averaging algorithm, “leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates.” Each round, a server sends the current model to a sample of clients; each client trains on its own data and returns a weight update; the server averages the updates into a new global model. The paper’s engineering contribution was making this cheap enough to be practical, cutting required communication rounds by 10 to 100 times compared to naive distributed stochastic gradient descent.

The same pattern now runs in two distinct settings. Cross-device federated learning spans millions of consumer endpoints, as in the original mobile-keyboard use case. Cross-silo federated learning spans a handful of institutions, such as hospitals or banks, that cannot pool records because of confidentiality obligations, competition law, or data-transfer restrictions. The privacy argument is the same in both: the training data never moves.

The protection you get by default

Three concrete benefits follow directly from the architecture, before any cryptography is added.

Data minimization by design. The coordinating server never holds the raw records, so there is no central corpus to breach, subpoena, misuse internally, or repurpose later. For a compliance reader, this maps cleanly onto the minimization and storage-limitation principles: the controller running the aggregation server processes derived updates, not the underlying personal data.

A smaller breach surface. A compromised central training pipeline in conventional machine learning exposes the entire pooled dataset. In a federated design, the same compromise exposes model updates, which are far harder to exploit, and each participant’s raw data remains behind that participant’s own controls.

Feasibility of collaboration that would otherwise be unlawful or refused. Cross-silo participants can produce a jointly trained model without any of them disclosing records to the others. This is why the technique features in financial-crime and public-health pilots: the alternative, centralizing everyone’s data, is often the thing the law or the data-sharing agreement forbids.

What model updates still leak

The architecture stops direct disclosure. It does not stop inference, and this is where much of the vendor marketing quietly overstates the protection. NIST’s joint UK-US blog series on privacy-preserving federated learning states the problem plainly: “Model updates are determined by the training data. In some cases, it’s possible to recover information about the training data from the model updates.” The final model carries the same risk, because it too reflects what it was trained on.

The strongest demonstration is Deep Leakage from Gradients by Zhu, Liu, and Han. Given the gradients a participant shares in collaborative training, their optimization attack reconstructs the underlying training inputs, and the recovery is “pixel-wise accurate for images and token-wise matching for texts.” In other words, an adversary who observes an individual client’s raw update, whether that adversary is a malicious server or an eavesdropper, can in some settings read back the training data itself. Membership inference is the quieter cousin of this attack family: instead of reconstructing a record, the adversary confirms a specific record was in the training set, which is covered in detail in our explainer on membership inference attacks. Real-world attacks on deployed ML systems, federated or not, are tracked at ai-alert.org.

The operational conclusion: plain federated learning protects against a curious aggregator reading raw data, but not against an aggregator or observer analyzing individual updates.

The add-ons that turn it into a privacy guarantee

Because of those attacks, serious deployments layer additional controls on top. NIST and its UK counterparts call the combined stack privacy-preserving federated learning, and it was the subject of the joint US-UK PETs Prize Challenges.

Secure aggregation answers the malicious-server problem with cryptography. The protocol from Bonawitz et al. lets the server compute only the sum of client updates while learning nothing about any individual contribution, tolerates up to a third of clients dropping out mid-protocol, and does so at a communication overhead of roughly 1.73x to 1.98x at realistic scales. With secure aggregation in place, the gradient-reconstruction attack loses its input: there is no individual update to invert, only an aggregate over many clients.

Differential privacy answers the trained-model problem statistically. Adding calibrated noise to updates or to the aggregation step bounds how much any single participant’s data can influence the released model, which limits membership inference and reconstruction from the final artifact. The cost is a privacy-utility trade-off that must be tuned per deployment, which is exactly why the PETs challenges asked teams to deliver “end-to-end privacy guarantees” rather than a single mechanism.

The stack matters more than any layer. Federated learning limits collection, secure aggregation blinds the server, and differential privacy bounds what the outputs reveal. Each addresses a failure mode the others do not.

What this means for a compliance program

Treat “we use federated learning” the way you would treat “we use encryption”: as the start of the diligence conversation, not the end. Questions worth putting to a vendor or an internal ML team: Does the server ever see individual updates, or only secure aggregates? Is differential privacy applied, and at what parameters? What is the threat model, an honest-but-curious server, a malicious server, colluding participants? Whether model updates constitute personal data in a given deployment turns on identifiability in that deployment, and the leakage research above is the reason a blanket “it’s anonymous” answer should not survive review. Regulatory expectations for privacy-enhancing technologies are still forming; ongoing rulemaking and guidance on AI systems is tracked at neuralwatch.org.

Federated learning is one of the few privacy technologies that changes the architecture rather than the paperwork. Deployed with secure aggregation and differential privacy, it can carry real weight in a data-protection design. Deployed bare, it moves the leak from the dataset to the gradients.

Related across the network

• Membership Inference vs. Model Inversion: Privacy Attacks — adversarialml.dev
• Model Inversion Attacks: Reconstructing Training Data from Output — adversarialml.dev
• Training Data Extraction from LLMs: The Carlini Results Explained — adversarialml.dev
• TensorFlow Security Vulnerabilities 2026: CVEs and Supply Chain — mlcves.com
• Privacy-Preserving ML in Production: What Substra Actually Does — sentryml.com