Benefits of signing RSA-encrypted data?

Question

I recently came across this (PDF) journal article, which describes what it calls "A New Design Of A Hybrid Encryption Algorithm" (which I must say, doesn't sound very novel, even in 2014!):

1. An AES key K is chosen

2. Encrypt message M using AES and key K

   eM = AES-encryption(M)

3. AES key K is encrypted using RSA with the public key of the receiver

   eK = RSA-encryption (K)

4. The cipher text eM is fed to SHA1, which generates a 160-bit message digest

   mD = SHA1 (eM)

5. The message digest is signed using RSA with the private key of the sender

   DS = RSA-sign (mD)

6. The encrypted message eM, digital signature DS and AES encrypted key eK are transmitted to the user over a network

So, basically standard AES-RSA hybrid encryption, but with the additional of an RSA signature (a signed message digest) that is intended to provide message authentication.

My question is whether signing the ciphertext actually provides any extra security (i.e. integrity or authentication) - doesn't encrypting the AES key with the receiver's RSA public key effectively provide message authentication?

Answer 1

Yes, signing the ciphertext does provide extra security, and yes, that would be integrity and authenticity of the message.

Encrypting with a public key does not provide integrity / authenticity against active attacks. That's kind of obvious: an adversary can simply encrypt another message with the public key - it is public after all.

The scheme however uses asymmetric encrypt-then-sign. This is dangerous as an adversary could strip off the signature to replace it with its own. That way it looks like the adversary has signed / send the message. However, this is only possible if the public key of the adversary is trusted within the system. More details are (indeed) here.

It will also be impossible to verify the message afterwards (unless you keep the ciphertext message as well, and make sure that the plaintext message and ciphertext message are irreversibly linked).

So when used correctly this kind of scheme provides integrity / authenticity. But it also has known limitations; that is: known to the cryptographic community, not the authors.

---

These kind of papers are used by the writers to get their degrees or progress within their studies. They provide zero insight and contain clear issues:

• no mode of operation is defined for the symmetric cipher, no IV handling is performed;
• the padding modes of RSA are not specified, not for encryption nor for signature generation ... there is only a limited description of the textbook RSA operation;
• the hashing is not thought of to be part of the signature generation function;
• the use of SHA-1 and the references to MD5 and blowfish, years and years after SHA-2 has been standardized.

And quite obviously there is no security analysis at all, which makes the entire paper a waste of bits and - worse - time.

The given "protocol" (and the quotes are needed here) could easily be extended with a message nonce and sender/recipient, making it more secure.

Answer 2

My question is whether signing the ciphertext actually provides any extra security (i.e. integrity or authentication) - doesn't encrypting the AES key with the receiver's RSA public key effectively provide message authentication?

With asymmetric encryption, you need to consider two different kinds of authentication of the message:

• Sender Authentication. That is, the fact that the recipient can be sure who sent a given message. For this you actually need signatures (or MACs with pre-shared keys), as without a signature, an attacker could just intercept a ciphertext, drop it, and encrypt a different message and send that.
• Recipient Authentication. That is, the fact that you can actually, as a sender, be sure that that the message reaches the intended recipient. You essentially get this for free with hybrid encryption.

Now to a side-note: The given mechanism signs the ciphertext. It is preferable to sign the plaintext, append the signature and encrypt the combination. For details, please see this question: "Should we sign-then-encrypt, or encrypt-then-sign?".

---

As to how ideally do things, let's go through this: Let $H$ be a random oracle (e.g. instantiated by SHAKE128) with output size 224 bit. Let $(n,e)$ be a RSA public key, let $\operatorname{sign}_{sk}(m)$ be any EUF-CMA secure signature scheme (eg ECDSA). And finally let $E_{K\parallel IV}(M)$ denote AES-128-GCM encryption using $K$ and $IV$, returning $(c,\tau)$. Then the following is the "proper" way to encrypt and sign a message $m$:

1. $r\xleftarrow{\$}\{0,...,n\}$, that is, sample an integer smaller than $n$, uniformly at random and call it $r$.
2. $c_1\gets r^e\bmod n$, that is, text-book RSA encrypt the random value and call the result $c_1$. This is secure because we are doing RSA-KEM here.
3. $m'\gets (m,\sigma=\operatorname{sign}_{sk}(m))$, that is, sign the message, call the signature $\sigma$ and pair it with the message (appending suffices if $\sigma$ has a known length) and call the result $m'$.
4. $k\parallel IV\gets H(r)$, that is, apply the key derivation function / random oracle / hash function to get the key and IV for the symmetric encryption.
5. $c_2\gets E_{k\parallel IV}(m')$, that is encrypt and symmetrically authenticate the message.
6. return $(c_1,c_2)$, that is, return a pairing of $c_1$ and the encrypted message. Simple concatenation suffices if $c_1$ is left-zero padded to the same length of $n$.

Answer 3

Your design is exploitable to MITM (man-in-the-middle) attack, an attacker can intercept your message and deliver "personal tampered message" encrypted with receiver's public key.