Is the Rényi entropy a continuous function with respect to the parameter $\alpha$?

Question

The Rényi entropy of order $\alpha$, where $\alpha > 0$ and $\alpha \neq 1$, is defined as $$ \mathrm{H}_\alpha(X)=\frac{1}{1-\alpha} \log \left(\sum_{i=1}^n p_i^\alpha\right) $$ Here, $X$ is a discrete random variable with possible outcomes in the set $\mathcal{A}=\left\{x_1, x_2, \ldots, x_n\right\}$ and corresponding probabilities $p_i \doteq \operatorname{Pr}\left(X=x_i\right)$ . I would like to know if $\mathrm{H}_\alpha(X)$, considered as a function of $\alpha$, is a continuous function.

I think that, if the probability distribution function $p_i$ is $\neq0$ for all $i$, then $\mathrm{H}_\alpha(X)$ will be a continuous function of $\alpha$. This is because the sum and logarithm operations in the entropy formula are continuous functions, and the composition of continuous functions remains continuous. Is it true also if $p_i=0$ for some $i$?

Answer 1

Yes it is continuous for the case $p_i\neq 0,$ as you pointed out.

In information theory, including for Shannon entropy, quantities such as $$ p_i \log( 1/p_i) $$ are continuous on $p \in (0,\infty)$ and by convention are taken to obey $$ p \log(1/p)\rightarrow 0,\quad \textrm{as}~p\rightarrow 0. $$ The Renyi entropy $H_\alpha(X)$ is defined and continuous for $\alpha \in [0,1)\cup (1,\infty).$ Its problem regarding discontinuity is not $p_i=0$ but the value of $\alpha=1.$ Rewriting $H_\alpha(X)$ with the constant $\alpha/(1-\alpha)$ in front makes this clear since the quantity inside the log is now simply the $\alpha$ norm of a probability distribution, i.e., $$ \left(\sum_i p_i^\alpha\right)^{1/\alpha} $$

which is finite for $\alpha$ restricted to the domain above.

Also, the Renyi entropy actually converges to the Shannon entropy as $\alpha\rightarrow 1.$

Answer 2

I would refer you to the paper Rényi Divergence and Kullback-Leibler Divergence, by Tim van Herven and Peter Harremoës. They study the properties of the Rényi divergence, a quantity representing a kind of "distance" between probability distributions. More precisely, for a positive order $\alpha\neq 1$, the Rényi divergence of probability distribution $P=(p_1, \dots, p_n)$ from probability distribution $Q=(q_1, \dots, q_n)$ is defined as $$D_{\alpha}(P\|Q) = \frac{1}{\alpha-1}\log\left(\sum_{i=1}^n \frac{p_i^{\alpha}}{q_i^{\alpha-1}}\right),$$ where by convention $0/0=0$ and $x/0=\infty$ for $x>0$.

They show in particular (see Theorem 7), that the Rényi divergence is continuous in $\alpha$ on $\mathcal{A}=\{\alpha\in[0, \infty]|0\leq\alpha\leq1 \text{ or }D_{\alpha}(P\|Q)<\infty\}$ (notice how the extended orders $0, 1$ and $\infty$ are included). Since the Rényi entropy of a probability distribution $P=(p_1, \dots, p_n)$ can be expressed in terms of the Rényi divergence of $P$ from the uniform distribution $U=(\frac1n, \dots, \frac1n)$ as $$H_{\alpha}(P)=\ln(n)-D_{\alpha}(P\|U),$$ the continuity argument also holds for the Rényi entropy.