Public key encryption schemes and security of an uncommon use

Question

Suppose that we use a public key encryption scheme by keeping the public key secret. Can we consider a public key encryption scheme at least as secure in this private-key setting, when compared to the normal use?

Answer 1

Assuming that you're using the same definition of "secure" for both schemes¹, then yes, it is indeed trivially true that modifying a public key encryption scheme by restricting the distribution of the public key will yield a scheme that is "at least as secure" as the original.

Specifically, consider an attacker who wants to break the modified scheme. We don't really care what the attacker wants to do (although, usually, someone attacking an encryption scheme would want to extract some information from an encrypted message), just that it's something that the modified (and the original) scheme is designed to prevent. There are two possible cases:

• Either the attacker knows the public key, or can somehow determine it from other information that they have. In this case the two schemes are, from the attacker's viewpoint, equivalent, and therefore equally secure.

• Alternatively, the attacker may not know the public key. In this case, if they were given the public key, that would make the two schemes equivalent from their viewpoint. Since giving the attacker extra information cannot make the scheme harder to break, that means that the modified scheme cannot be easier to break than the original.

Therefore, in neither case can the modified scheme be easier to attack than the original.

It is, however, quite possible for the two schemes to be equally secure. For many real world public-key encryption schemes that is indeed likely to be the case, if only because such schemes are usually not designed to hide the public key in any way, and thus may allow attackers to easily learn it.

In general, public-key crypto is designed with the assumption that the public key is public, and trying to make it otherwise rarely if ever yields any significant benefit. While it may be possible to repurpose some crypto algorithms originally intended for public-key encryption for other uses, in some of which the "public key" might in fact need to be kept secret (as e.g. in this question I asked before), that generally invalidates much of their original security proofs and requires reanalyzing the whole algorithm from the ground up with the new assumptions on what's public and what isn't.

---

^{1) Specifically, the definition of what would constitute a successful attack should be the same for both schemes, and should not make reference to anything that would differ between the schemes. In particular, for the purposes of this argument I'm excluding "the attacker learns something that isn't already public" as an admissible definition of a successful attack, since the two schemes differ in what they consider to be public information.}