What is the best way to sign a message using RSA?

Question

I was preparing for competitive exam, and I came across this question. I had a lot of discussion with my friends before I resorted to posting it here and I wanted myself to be clear on this.

---

Let $M$ be a confidential email that Alice wants to send to Bob, $K_B$ be Bob’s encryption public key, and $K_A^{-1}$ be Alice’s private key for signing. Which of the following options would be the best choice for protecting confidential emails?

(A) Send $E_{K_{B}}(M),Sign_{K_A^{-1}}(K_B)$

(B) Send $E_{K_B}(M),Sign_{K_{A}^{-1}}(M)$

(C) Send $E_{K_B}(M),Sign_{K_{A}^{-1}}(E_{K_B}(M))$

(D) Send $E_{K_B}(M),Sign_{K_{A}^{-1}}(K_{A}^{+1}(M))$

---

I marked (B), but correct answer is given to be (C).

I understand the first part, that we are encrypting the message with Bob’s public key, but my understanding says, that we sign the message using Alice’s private key and send it to Bob along with the encrypted message.

But seems like option (C) says that along encrypted message, Alice should sign the encrypted message and then should send this signature along with the encrypted message to Bob.

My query is as per option (C), Why are we signing the whole encrypted message instead of the original message as shown in (B).

Answer 1

Option A signs the public key rather than the message. As the goal is to authenticate the message, the signature over the public key is useless for this purpose.

Option B signs the message. It however has the disadvantage that it is possible for an adversary to guess the message and verify the correctness of the guess. As such, it doesn't provide full confidentiality. It may also be vulnerable against oracle attacks unless a IND_CCA2 secure cipher is used; it could be that the receiver leaks information when trying to decrypt unauthenticated ciphertext.

Option C signs the encrypted ciphertext. It doesn't provide any authenticity of the message for email encryption because somebody can simply strip off and replace the signature. It would only provide authenticity if Bob only expects signatures of Alice, and disregards any other public keys that can be used to verify the message. This is however commonly not the case for person-to-person mail encryption.

Option D is clearly some mumbo jumbo to have a fourth option. As the operation $K_{A}^{+1}(M)$ doesn't make any kind of sense even with regards to the notation, it is probably best to ignore it.

---

Option B is an encrypt-and-sign scheme, option C is an encrypt-then-sign scheme.

Generally we use a (missing) sign-then-encrypt scheme where we send $E_{K_B}(M, \text{Sign}_{K_{A}^{-1}}(M))$. Obviously sign-then-encrypt should be used with care as well; it does require a IND_CCA2 secure cipher just like option B. Note that I've left out some kind of encoding function to merge the message with the signature.

Now that you know the terms you can look up Q/A and discussions such as the one here. Note that we often need to include additional information in the scheme for it to be secure in a practical sense; protocol design is however a separate topic.

---

EDIT: Come to think of it, it is also possible to replace the signature for option B, so that would make option C marginally better.

Answer 2

Most common symmetric-key modes of operations and AEAD algorithms operates in Encrypt-the-MAC fashion, so similarly, one better use encrypt-then-sign paradigm.

All of the choices correctly stated that one should use Bob's public key to encrypt, so the only question remains is how to sign.

Remove A from consideration immediately as you realize that protects not the integrity&authenticity of $M$ but Bob's public key. Also remove D from consideration as that's a bad case of reusing signing key for encryption.

Choice B may have some problem if the signature can be used to recover the message (e.g. Unhashed), so it may be bad for confidentiality in some cases.

So go with C.