Signing 20-byte message with 256-bit RSA key working with openssl.exe but not in code

Question

I have a 256-bit private key that I want to use to sign a SHA-1 digest (20 bytes). Using openssl directly it seems to work

echo doesntmatter | openssl dgst -sha1 -binary | openssl rsautl -sign -inkey 256bit_private_key.pem | openssl enc -base64

gives me a Base64 output as expected.

But doing it with the OpenSSL fails with "error:04075070:rsa routines:RSA_sign:digest too big for rsa key". As you can see below, I'm passing the 20-byte (SHA_DIGEST_LENGTH=20) SHA-1 digest as input to RSA_sign. Even with padding it shouldn't be more than the maximum of 32 bytes that I can encrypt with a 256 bit modulus key?!

unsigned char digest[SHA_DIGEST_LENGTH];
SHA1(message, messageSize, digest);

unsigned int privateKeySize = RSA_size(privateKey); // 256 bits = 32 bytes
unsigned char* signature = new unsigned char[privateKeySize];
unsigned int signatureSize;

int res = RSA_sign(NID_sha1, digest, SHA_DIGEST_LENGTH, signature, &signatureSize, privateKey);

if(res == 0)
{
    int err = ERR_get_error(); // 67588208
    char *s = ERR_error_string(err, 0); // error:04075070:lib(4):func(117):reason(112)

    delete [] signature;

    [...]
}

What am I doing wrong in the code?

Answer 1

check out this SO answer. rsautl is depreciated in favor of pkeyutl

Essentially, RSA signing should use RSA-PSS signature padding scheme to be secure, and the RSA-PSS will be strictly larger than the digest, because of the necessity to communicate a salt. additionally, RSA_sign does a digest as a part of the signing scheme, so your code is going to do a digest of a digest. you want to instead either pass the message in directly, or use RSA_private_encrypt combined with a suitable RSA_padding_add_x call.

and as I said in my comment, a 256-bit key is sufficiently secure for a symmetric algorithm, but it will be trivial to crack for an RSA private key (on the order of 4 minutes on a couple of year old machine). Using a bare minimum of a 1024 bit key (and probably a 2048 or 4096 bit key) will give you a security level roughly equivalent to a 128 bit symmetric algorithm.

Answer 2

Re Peter Elliott: not quite.

The PKCS1v1_5 standard sequence in RFC8017 section 8.2.1 (and 9.2, similar in rfc4347 and rfc3447) is hash, encode in ASN.1, pad ('type 1') and modexp d. (PSS omits the ASN.1 step and uses different padding.)

OpenSSL (low-level) RSA_sign does the last three (with v1.5 padding) but not the first -- i.e. it encodes the hash but it doesn't do the hash; even-lower RSA_private_encrypt does the last two. The higher-level envelope module (original EVP_Sign{Init,Update,Final}*, enhanced EVP_DigestSign*) does all four. It's the ASN.1 encode that made the value too large for a 256-bit key -- which as stated is breakable anyway.

rsautl does only the last two steps not the encode, so it runs but gives a nonstandard result. pkeyutl for RSA does the same by default but can be told to do the encode.

Also, PSS has a better security proof, but as far as I've heard there's no actual attack on v1.5 and it's still widely used. If you have the choice of PSS (both/all parties support it) choose it, but don't feel worried about using v1.5.

Answer 3

Guess I found the solution. openssl rsautl -sign uses RSA_private_encrypt instead of RSA_sign (what I would have expected). RSA_sign creates a longer structure than the 20-bytes message I provided, so it fails with the given error.