2

The question is just for fun, and I feel like I'm missing a clever way of thinking about it.

suppose that you are on an alien planet, and you are trying to learn their language. You break into one of the aliens houses and get on his computer and print the contents of a file by accident. you have to figure out if this document is written in their language, or if it's just a binary file.

the idea here I think is that the alphabet for the language is smaller than the alphabet for a printed binary file. You would expect many more repeats from the string with the smaller language. after some string length, you would be able to give a pretty good estimate of the number of letters in the alphabet from which the text was written.

anyway, the question is, what is the probability of a string of length $n$ chosen randomly with an alphabet of size $m$ will have $k$ unique letters?

ZKe
  • 800

3 Answers3

3

The number of words of length $n$ that use exactly $k$ letters from an alphabet of size $m$ is

$$ {m \choose k} k! \, S_{n,k} = \frac{m!}{(m-k)!}S_{n,k}$$

where $S_{n,k}$ is the Stirling number of the second kind, (you can think the alphabet as a set of $m$ urns, and $n$ numbered balls, each one corresponding to a position in the word). The total number of words is $m^n$. Hence the probabilities are given by

$$ p_k^{n,m} = \frac{m!}{(m-k)!} \frac{S_{n,k}}{m^n}$$

leonbloy
  • 66,202
  • yeah, this is good. $S_{n,k}$ is unlabelled partitions, the $m\choose k$ gives all the label sets and the $k!$ gives all the labelings for each of those sets. This is a good formula to know. – ZKe Jun 02 '13 at 04:13
2

The expected number of distinct letters is easy to compute. The distribution of the number of distinct letters is much less pleasant, and I do not know of anything that avoids messy Inclusion/Exclusion.

Let the letters be $a_1,a_2,\dots,a_m$. For any $i$, let $X_i=1$ if the letter $a_i$ occurs in our text, and let $X_i=0$ otherwise. Then the number of different letters is $X_1+X_2+\cdots +X_m$. By the linearility of expectation, we have $$E(X_1+X_2+\cdots+X_m)=E(X_1)+E(X_2)+\cdots +E(X_m).$$ To find the expectation of $X_i$, we find the probability that $X_i=1$. This is $1-\left(\frac{m-1}{m}\right)^n$. So the expected number of distinct letters is $$m\left( 1-\left(\frac{m-1}{m}\right)^n \right).$$

André Nicolas
  • 514,336
  • Yeah it's the distribution that was giving me trouble. Although this is a nice way of thinking about the expectation. – ZKe Jun 02 '13 at 03:32
0

I think you are missing a great deal of information thinking only about the number of distinct characters. A pure text file of English will have only about 60 of the 256 bytes represented, but 256 seems a natural number to use and maybe their language fills it up better. On the other hand, maybe their byte is only 6 bits. But the distribution of those characters will be very representative of text. exe files will have a very different distribution, but it will not be random either as certain instructions are much more common than others. Compressed files are probably much closer to even in the distribution of bytes-that is what makes compression work. I think it would be easy to distinguish the population of text files from exe files from compressed files. Whether you could tell the first two apart seems more questionable.

Ross Millikan
  • 383,099
  • these are very tidy aliens, all of their non-text files are perfectly compressed. – ZKe Jun 02 '13 at 03:49
  • @Zackkenyon: Then it should be easy to find the text files. A perfectly compressed file has all the bytes with equal probability (the little stuff at the beginning to specify the compression fades into nothingness). Any file that doesn't satisfy that is a text file. – Ross Millikan Jun 02 '13 at 03:55
  • keep in mind that these "files" I'm looking at are extremely finite in size. This is exactly the statement that I am trying to probabilistically extract from my observation. – ZKe Jun 02 '13 at 04:04
  • @Zackkenyon: the point is these distributions are extremely different. It won't take much volume to tell one population from the other. In Cryptonomicon the hero has reduced the decryption of some text to a brute force search over an accessible number of possibilities, but the computer has to find the correct one. He just checks the letter frequencies of the output. If the key is incorrect, the frequencies will be badly wrong. – Ross Millikan Jun 02 '13 at 04:15
  • yeah but your confidence obviously increases with the length of the message. The question is still one of "how long do these strings need to be before I can expect my observation to be useful." – ZKe Jun 02 '13 at 04:30