Textbook RSA Attacks

Question

Given textbook RSA encryption, if an attacker obtains the ciphertext and has the public key that was used, can he or she decrypt said ciphertext without calculating the private key?

Answer 1

Without any additional knowledge? No.

The problem here is, that this understanding of security is wrong / too weak. Ciphertext-only attacks is the weakest kind of attack model, which is only relevant for classical ciphers any more.

To consider a cryptosystem secure today, it has to resist the much stronger attacks, most commonly the chosen-plaintext attacks and (in the case of public key crypto) chosen ciphertext attack. And on the other hand, we also don't require the attacker to decrypt the actual message. We consider the system broken if the attacker can distinguish messages already, for instance ciphertext indistinguishability.

... without calculating the private key?

If the attack goal would be decryption, the only known way is to break the private key. It is unknown, if the RSA problem and the factorization problem are equivalent or not. And if we had an algorithm which could decrypt but not calculate the private key, this question would be solved.

Answer 2

No, in general. But since in textbook RSA you do not use pad, you can have an attack better than brute force (under some plausible conditions). Say $c=RSA_{e}(m)=m^{e}\pmod n$ and $N$ is the number of bits of the message $m$ (i.e. $m\approx 2^N$). With some ("large?") probability $m=m_1m_2$ and $m_1,m_2<2^{N/2}.$ So $c/m_1^{e}=m_2^{e}\pmod{n}.$ The idea is to construct two lists (in ${\bf Z}_n$) $$L_1 =\{c/1^{e},c/2^{e},c/3^{e},\dots,c/2^{eN/2}\} $$ $$L_2 = \{1^{e},2^{e},3^{e},\dots,2^{eN/2}\}$$ An element of the intersection, $c/a^{ei}=b^{je}$ provides $m_1=a^{i},m_2=b^{j},$ thus $m=a^{i}b^{j}.$ Since construction,sorting and finding the intersection takes $\tilde{O}(2^{N/2})$ time, you get a better time than brute force.

Now, if $m$ has 64-bit, you can experimentally see that with probability $\approx 1/4$ you get that $m=m_1m_2$ with $m_1,m_2<2^{32}.$ For larger messages this probability is smaller.

So, the answer to your question is that you can decrypt without knowing the private key, with some large probability if the message is small enough.