What was the motivation for the Serpent linear function?

Question

The design of the Serpent cipher (state of 4 32bit words) is odd in some ways and the AES submission document doesn't shed light on the oddities.

Firstly, it almost tries an ARX approach except that there is no addition present. Just linear mixing. Was ARX unknown circa 1997? Some other AES submission did have addition and even multiplication.

Then there is the oddity that it performs mixing between words in the linear layer. Why do this when the s-box is performed across the 4 words, this mixes the 4 words together. You then only need to mix bits together in each of the 32bit words.

AES Rijndael mixes bits in each 32 bit word, then transposes 8bits from each word for the next round. Another modern cipher which has something similar is Ascon, it mixes bits within each word and then applies an s-box across words.

Answer 1

I cannot answer what was real motivation of Serpent's design, because I did not design it. I can speculate.

Serpent uses S-boxes for non-linearity therefore it only needs linear mixing.

ARX as basic concept was known at the time (RC2, TEA, ...). However ARX was not well researched at the time and had reputation of being weak. Alternating between XOR and addition was often used just as ad-hoc additional protection.

Serpent's designers have chosen to make design similar to DES and DES does not use addition. Addition is also not hardware friendly.

Linear mixing could not really be implemented efficently without mixing between words, because Serpent only has 4 words. It could be done like in Rectangle cipher by using just rotations. This however would be much weaker round function that would probably not work well with Serpent's expensive S-boxes. Even though S-boxes and linear layer difuse between same words, linear layer rotates words, so independant S-boxes get mixed. This is nothing odd. I believe Serpent's designers also did not want cipher to have aligned structure (at a time this was concern for some with Rijndael).

I think it is also mentioned somewhere that shift instead of rotation is also used to make sure that characteristic does not survive indefinitely.