0

I'm trying to understand how to figure out how much information is lost in going from n dice to the total score.

If I rolled n-dice I could add up their face values to find the total sum, however if I just take the total sum I most likely can't tell which dice had which face value (except for all 1's or 6's). That is, such as the sum of 3 dice to be 11, just knowing 11, I can't tell exactly how much each die contributed to the sum. So it seems as if there is a loss of information in going from knowing the faces of the die to just knowing the total score.

Now, from A Webb's answer here it is simple to find the amount of information in n dice rolls:

$$- \log_2{((\frac{1}{6})^{n})} = - n * \log_2{(\frac{1}{6})} = n * 2.58$$ bits of information. As the probabilities follow a uniform distribution and all face probabilities are the same we'd expect the average information of each roll to be the same as each of the individual events. So the entropy (average information) for each die roll 2.58 bits. We can then sum this entropy $n$ times for $n$ dice.

Next, as the expected value of the sum of random variables is the sum of their expected values and the expected value for each die is $\frac{1+2+3+4+5+6}{6} = \frac{21}{6}$ we have that the expected value of the sum of n dice/rolls to be $n * 3.5$. I however am not sure where this gets me (if anywhere) in finding the information entropy of the sum of scores so as to compare that to that of the entropy of the n dice.

Brian Droncheff
  • 1
  • 1
    The expected value of the sum has nothing (directly) to do with the entropy of the sum. You need to compute the probability mass function of the sum (i.e., identify which values the sum can take and what are the corresponding probabilities) and apply the standard entropy formula – Stelios Jan 24 '21 at 22:26
  • @Stelios. I understand needing to compute the probability mass function. Intuitively, because of the symmetry of the expected values of the sum, I was thinking that there was a way to take the expected value of the sum of rolls of the dice $n * 3.5$ and find the entropy this way which is what I was trying to get at. Glad to know it isn't really useful but then am wondering how one figures out this probability mass function for a discrete normal distribution without doing it value by value. – Brian Droncheff Jan 24 '21 at 23:13
  • 1
    there is no way around it. You have to do it value by value. However, for large $n$, you might use the central limit theorem as an approximation – Stelios Jan 25 '21 at 06:31

1 Answers1

1

As noted in the comments, you need to compute the entropy of the sum numerically. By analogy with the result for a binomial, you might assume that for large $n$

$$H(S) \approx \frac1 2 \log_2 \big( 2\pi e\, \sigma^2 \big) = \frac1 2 \log_2 \big( 2\pi e\, \frac{35}{12} \big) + \frac12\log_2(n) = 2.819256 +\frac12\log_2(n) \tag 1$$

For example, for $n=5$ we get $H(S) \approx 3.9802$ while the true value is $3.9774$

For $n=9$ the approximation gives $4.4042$ , vs the exact $4.4035$

Hence, $(1)$ seems a good approximation, and the information loss is big: the entropy of the die is linear in $n$, while the entropy of the sum grows logarithmically.

leonbloy
  • 66,202
  • Thanks, very appreciated. I'm wondering, how did you get $35/12$? From the other question it looks like $\sigma^2 = np(1-p)$ so we'd have $35/12 = p(1-p)$ though I'm not sure how this is obtained. – Brian Droncheff Feb 02 '21 at 22:11
  • $35/12$ is the variance of each die https://en.wikipedia.org/wiki/Discrete_uniform_distribution Hence the variance of the sum is $n 35/12$ – leonbloy Feb 02 '21 at 23:38
  • I'm a little lost on how you get the extra $n$, meaning that, the variance for an n-sided die is $\frac{n^2-1}{12}$ which for 6-sides we'd have the $\frac{35}{12} = \sigma^2$, so we'd just have $\frac{1}{2}log_2(2\pi e\sigma^2) = \frac{1}{2}log_2(2\pi e\frac{35}{12})$, so where does the other $n$ come in? – Brian Droncheff Feb 03 '21 at 03:28
  • Also, so that I'm clear, $n$ in your answer corresponds to the number of dice rolled? If so then how did you figure out the actual value since, as you mentioned, you would have to compute the entropy of the sum numerically in which case you would have possible sum values of 5 * 1 = 5 to 5 * 6 =30 for the six sided die. I'm guessing you didn't do this by hand. – Brian Droncheff Feb 03 '21 at 04:01
  • Also, as a side note, you can upper bound the entropy of the sum by log(n+1) simply because of the support size. So, you don't need to compute the entropy of binomial if all you care is observing the linear vs logarithmic growth situation. – E-A Mar 10 '21 at 20:15