1

Given a discrete random variable $X$, which takes values in the alphabet $\mathcal {X}$ and is distributed according to $p:{\mathcal {X}}\to [0,1]$ the Shannon entropy is defined as: $$\mathrm {H} (X):=-\sum _{x\in {\mathcal {X}}}p(x)\log p(x)$$ As we know (see e.g. Prove the maximum value of entropy function) the maximum value of the Shannon entropy is $\ln N$ where $N=\operatorname{card}(\mathcal X)$.

The Rényi entropy of order $\alpha$, where $\alpha \geq 0$ and $\alpha \neq 1$, is defined as $$\mathrm {H} _{\alpha }(X)={\frac {1}{1-\alpha }}\log {\Bigg (}\sum _{i=1}^{N}p_{i}^{\alpha }{\Bigg )}$$ My question is: is it possible to find an upper bound also for the Rényi entropy?

Mark
  • 7,702
  • 6
  • 41
  • 80

1 Answers1

1

From the Wikipedia page:

The Rényi entropy for any $\alpha \geq 0$ is Schur concave.

This implies that the maximum is still achieved for the uniform distribution, i.e., $$ H_\alpha(X) \leq \frac{1}{1-\alpha} \log \!\left(\sum_{i=1}^N \frac{1}{N^{\alpha}}\right) = \log N $$ for any $\alpha \geq 0$ and $X$ supported on a domain of size $N$.

Another way to find this upper bound (which one can then check is achieved for the uniform distribution in the domain) is to use the fact that $H_\alpha(X)$ is a non-increasing function of $\alpha\geq 0$, so $H_\alpha(X) \leq H_0(X) = \log N$.

Clement C.
  • 68,437
  • Thank you! I had never heard of Schur-convex/concave functions. How this implies that the maximum is still achieved for the uniform distribution? – Mark Jan 10 '23 at 11:17
  • @Mark Schur-concavity of $f$ implies that if you have a vector of probability $x$ majorizing $y$, then $f(x)\leq f(y)$. Majorization of the uniform distribution $y$ by any other distribution $x$ is immediate from the doubly stochastic matrix characterization (see linked page on Wikipedia in the previous sentence): take this double schotastic matrix $D$ to be the (scaled by $1/N$) all-ones matrix. – Clement C. Jan 10 '23 at 11:29