Why should CAST5 and 3DES not be used for encrypting files over 4GB?

Question

On https://www.gnupg.org/faq/gnupg-faq.html#define_cast, it says that

Like 3DES, its 64-bit block size means it should not be used to encrypt files larger than 4Gb in size…

How and why does the block size affect the security of the cipher?

Answer 1

If you use a key for close to $2^{n/2}$ blocks in CBC mode, then the chance of getting a collision in the ciphertext is getting rather high because of the birthday paradox. As the ciphertext is used as a vector for the next calculation, and since that vector should be unpredictable, you would likely lose confidentiality.

Note that the author seems to have put his mark at about $1 / 148$ as the chance for a collision to occur (see the calculation by fgrieu in the comments). That may be either too low or too high depending on the context. That is if bytes are meant instead of bits, otherwise it would be $1/9445$ - a somewhat more sensible value most of the time, but you would be left with only 512 MiB for a single key.