Are transposition ciphers insecure?

Question

Given plaintext: P[i], i = 1..N

Given stream of cryptographic numbers S[i, j], i = 1..N, S[i, j] in [1, j] generated by RC5 for example

We encrypt using Fisher-Yates shuffle algorithm:

for (int i = N, i > 1 ; i--) swap(P[i], S[i, i])

How to estimate complexity of breaking such chipher?

Is it lower than strength of S[i] generator?

How does it depend on value of N?

How does it depend on data we encrypt?

Unfortunately I was unable to find anything useful on topic.

P.S. Why not AES? Because chipertext is harder to detect using frequency analysis.

Answer 1

This is not very secure. You directly leak the symbol distribution, because only the order of symbols changes. For short enough messages this allows easy decryption – e.g. "dr olllWeoH" is quite clearly "Hello World". Even for long messages or binary values, the fact that you leak e.g. a crucial byte may be enough.

You also have not defined how the same key would be used to encrypt multiple messages. Using the same stream of random numbers would be fatal, allowing easy known-plaintext attacks at the very least. You would need an IV/nonce of some kind, which may itself make the fact that the message is encrypted detectable.

If the problem you are trying to tackle is hiding the fact that the message is encrypted, you should look into either steganography or format-preserving encryption:

• Steganography is about hiding secret messages in normal-looking messages. You could use a standard encryption algorithm (say AES-GCM) to encrypt the message and then hide it in a cover message using some steganographic scheme.

• Format-preserving encryption, on the other hand, allows you to keep the encrypted message in the same format as the plaintext, which might be sufficient to hide the fact that it is encrypted. For example, database fields can be encrypted in place. Beware that depending on the application FPE is not necessarily as strong as normal authenticated encryption can be.