what does differential privacy (in machine learning) promise or guarantee?

Question

I am recently reading some papers about privacy-preserving machine learning.

Some works incorporate the idea of differential privacy to protect the privacy of the training dataset when the model is published. The basic framework is as follows:

A trusted party (call it TP) possesses the training dataset in cleartext. TP then conducts some machine learning task and produces a prediction model. In the training process, TP will add some noise to the parameters (details omitted) such that the resulting model is noisy but is still accurate enough. It is then theoretically proved that some notion of differential privacy is achieved.

Meanwhile, I am aware of some "model inversion attacks" that can recover the training records given only a particular label and black-box access to the model.

So, what does differential privacy promise or guarantee in the context of privacy-preserving machine learning?

Specifically, suppose I am the owner of one record in the training dataset. The TP published a noisy model and tells me that it has $\epsilon$-privacy and the publication of the model will not violate my privacy. So what does it mean? Because I know that some model inversion attacks can recover my record. Does this violate the $\epsilon$-privacy?

Answer 1

Model inversion attacks are, by definition, impossible if the model generation process is $\epsilon$-differentially private for a sufficiently small $\epsilon$: differential privacy guarantees that if your original data was completely different, you would not be able to tell the difference just by looking at the model. Your data is effectively indiscernible from noise.

That said, the strength of the guarantee depends on the value of $\epsilon$, you can read a short explanation of that on this other question. If $\epsilon$ is very large, model inversion attacks might be successful in retrieving noisy or probabilistic information. I'm not aware of any attempt to experimentally verify for which parameters of $\epsilon$ it starts becoming a problem.