In short, short keys are susceptible to a certain class of attacks, key information over 2048 bits is just getting discarded, and exotic keys (those not aligned to bytes) are really either getting aligned to bytes by your implementation, or are just a very bad idea, depending.
RC4 is a fairly straightforward algorithm, let's walk through initialization to see what happens.
First, prep an incremental array of 256 bytes, say:
sbox[]
for i in 0 to 255:
sbox.append(i)
Then take some key k and prep a separate array key_array of 256 bytes by just cycling that key to fill it up.
key_index = 0
while len(key_array) < 256:
key_array.append(key[key_index])
key_index = key_index + 1
if key_index > len(key):
key_index = 0
So anything over 2048 bits (ie, 256 bytes) gets discarded.
Now you're going to walk through these two arrays, stepping from 0 to 255, summing what you find at that index in each array, and adding that sum to a counter (mod 256). You'll then swap two values in your sbox, using that running counter as the index for one value, and your current index step as the other.
for i in 0 to 255:
j = (j + self.sbox[i] + key_array[i]) mod 256
swap(self.sbox[j], self.sbox[i])
Now when you're looking at the first few i's, there hasn't been much shuffling yet. Those bytes may be shuffled again later, but if you can make a few assumptions, you may be able to detect a bias between the key and the first few bytes of the sbox after initialization.
It just so happens this attack works especially well with keys 5 Bytes and under, so that's probably where the admonishment to avoid keys under 40 bits came from. Funny enough, this attack doesn't just stop at five bytes though, the probability of success drops as the key size increases, until becoming essentially impractical for keys over 16 bytes.
As for exotic keys, most simple implementations will just take bytes. So if you provide 333 bits, it's probably just interpreted as a 336 bit (42 Byte) key anyway, perhaps with a few padded 0s.
If the implementation does not pad your input to the nearest byte, and simply cycles it throughout the 2048 available bits in the key array, then you may have effectively provided a 2048 bit key. As 333 shares no common factor with 8, it would not line up with an even byte until 2668 bits, well after our key array is full. Thus, save for patterns of repetition in your 333 bit key, you may have unintentionally generated the equivalent of a 256 byte key.
As a side point, 256 bytes of key sounds great, but it's actually not buying you as much entropy as you might think. The initialization of the sbox results in filling up 256 Bytes with information, and setting two counters, i and j. The key gives you the option of filling 256 Bytes, or 255 Bytes, or 254, and so on. So 256 might not necessarily be providing any additional entropy over a key of merely 180 Bytes.
Also, note that a lot of the research involves repeated key attacks, so as DW points out, it's really important to take some step to make sure you're not repeating keys. Rivest recommends taking the (key, initialization vector) and passing that through a hash function, so even a repeated initialization vector won't lead to any relationship between subsequent keys. Another area of research examines biases in the PRNG until it has "settled." Most implementations recommend adding another value to your initialization, a number of rounds of the PRNG to discard during setup (Rivest recommends discarding the first 256 bytes).
RC4 is one of the most simple, and thus one of the most well studied stream ciphers. It's hard to keep up with all the different classes of attacks, and determine which theoretical attacks have practical applicability. So don't take this as a guarantee that RC4 will forever be safe, keep checking back on the research.
Related key attacks against large keys:
Chen, J., Miyaji, A. A New Practical Key Recovery Attack on the Stream Cipher RC4 Under Related-Key Model. Lecture Notes in Computer Science (LNCS), v. 6584, pp. 62-76. 2011.
A. L. Grosul and D. S. Wallach, A Related-key Cryptanalysis of RC4. Technical Report-00-358, Rice University Department of Computer Science. 2000. (Note: 256 Byte keys especially bad.)
Attacks against small keys using the key scheduling algorithm:
Paul, G., Maitra, S. Permutation After RC4 Key Scheduling Reveals the Secret Key. LNCS, v. 4876, pp. 260-377. 2007.
Biham, E., Carmeli, Y. Efficient Reconstruction of RC4 Keys from Internal States. LNCS, v. 5086, pp. 270-288. 2008.
Akgun, M., Kavak, P., Demirci, H. New Results on the Key Scheduling Algorithm of RC4. LNCS, v. 5365, pp. 40-52. 2008.
Other weaknesses
- Fluhrer, S., Mantin, I., Shamir, A. Weaknesses in the Key Scheduling Algorithm of RC4, presented at the Eighth Annual Workshop on Selected Areas in Cryptography, 2001.
Rivest's defense of RC4
- Rivest, R. RSA Security Response to Weaknesses in Key Scheduling Algorithm of RC4. Tech Notes, RSA Laboratories. 2002.
