2

I'm writing a paper on Information Theory and I can't get my head around this task:

I'd like to prove that the maximum value of the Shannon entropy function $H$ is reached when each event in the set of probabilities $Ps=\{P(x_1), P(x_2), ..., P(x_n)\}$ has the same value.

The $H$ function is defined like this:

$$ H(S)=\sum\limits_{i=1}^{card(S)}P(x_i)*(-\log_2(P(x_i))) $$

I could only prove this with $card(S)<=2$ but could not find any technique to do it for $card(S) = N$.

I think that a possible solution would be solving it with a proof by induction using $card(S)$ (the length of $S$) as our parameter.

kodlu
  • 10,287
Liam Giannini
  • 123

1 Answers1

6

There are many ways to do this, but perhaps one of the easiest is via Jensen's inequality. Let $n$ indicate the cardinality of $S$. Then, write $$ \begin{align} H(S) & = \sum_x p(x) (-\log p(x)) \\ & = \sum_x p(x) \log \frac{1}{p(x)}\\ & \le \log \sum_x p(x) \frac{1}{p(x)}\\ & =\log n \end{align} $$ The inequality uses Jensen's inequality and that $\log$ is a concave function. It is easy to check that the uniform distribution, $p(x)=1/n$ for all $x$, achieves this upper bound.

Artemy
  • 1,283
  • Doesn't Jensen's inequality only works for $card(S)=2$ ? – Liam Giannini Mar 12 '20 at 20:02
  • 1
    By $card(S)$, do you just mean the number of outcomes of the random variable $S$? Then, no, Jensen's inequality holds for any $S$ of any finite or infinite cardinality. – Artemy Mar 12 '20 at 22:48
  • No, i mean the lenght of S – Liam Giannini Mar 12 '20 at 22:49
  • I'm sorry I don't understand what you mean by the "length of $S$", or perhaps your question at all. I think you need to clarify your question. What is the outcome space of the random variable whose entropy you are trying to compute? – Artemy Mar 12 '20 at 23:41
  • S is a set of events with random probability of happening, the sum of the probability of every event is 100% (1) and I know that the maximum value of the H function is reached when every probability is the same, so when $P(x_1) = P(x_2) = ... = P(x_n)$ the thing is that i don't know how to prove it – Liam Giannini Mar 13 '20 at 11:25
  • OK, then I think we are talking about the same thing. As I mentioned, Jensen's inequality works for any $n$. See e.g. https://en.wikipedia.org/wiki/Jensen%27s_inequality#Finite_form (in my answer, I use that $\log$ is concave) – Artemy Mar 13 '20 at 17:49