2

Let say we have a random variable $X$ with distribution $\mathbb{P}_X$. I would like to have a unique definition of entropy for discrete and random variable. According to this article of Wikipedia, https://en.wikipedia.org/wiki/Information_theory_and_measure_theory, I could define the entropy of $X$ relatively to a measure $\rho$ as $$ H_\rho(X) = - \mathbb{E}_{\mathbb{P}_X}\left[\log \frac{d \mathbb{P}_X}{d \rho}\right]$$ where $\rho$ is a measure on $Val(X)$, which could be either discrete or continuous, and $\frac{d \mathbb{P}_X}{d \rho}$ is the Radon-Nikodym derivative of $\mathbb{P}_X$ with respect to the measure $\rho$.

Then using either the counting measure in the discrete case or the Lebesgue measure in the continuous one, I can recover the definitions of Shannon and relative entropy. Am I correct ?

If yes then I have a problem because I could use the relative entropy between two measures $\mu$ and $\nu$: $$ D(\mu || \nu) = \mathbb{E}_{\mu} \left[\log\frac{d \mu}{d \nu}\right] $$ to define the entropy: $$H_\rho(X) = - D(\mathbb{P}_X || \rho)$$ But we know from Jensen's inequality that $D(\mu||\nu) \geq 0$ which would mean that $H_\rho(X) \leq 0$. There must be a mistake somewhere or something I'm missing but I can't find it...

PS : I know there is already a thread about this subject (Is there a unified definition of entropy for arbitrary random variables?) but it uses the definition of relative entropy from Gray which, from what I understand, is not exactly what I want.

MLass
  • 48
  • 1
    The differential entropy (and its variants) is not a "true" (Shannon) entropy. You cannot expect to find a useful+reasonable unified definition of entropy for discrete and continuous variables, because a (non degenerate) continous variable has infinite information content (hence infinite Shannon entropy). https://math.stackexchange.com/questions/2880612/comparing-differential-entropy-of-normal-distribution-to-shannon-entropy/2880947#2880947 https://math.stackexchange.com/questions/1398438/differential-entropy/1398471#1398471 – leonbloy Jun 07 '21 at 17:01
  • But isn't it a problem and a proof that the Shannon entropy is not a good object ? From my understanding, the relative entropy is a better one and I don't understand why we still use Shannon entropy or differential entropy. – MLass Jun 07 '21 at 21:16

1 Answers1

0

Do note that Jensen's inequality only works when your reference measure is a probability measure, and the proof of why KL is $\geq 0$ needs both measures to be probability measures. Otherwise, yes, KL divergence can be defined using Radon-Nikodym derivatives, as you outline.

E-A
  • 6,075
  • 1
  • 11
  • 21
  • Yes, I totally forgot this "detail"... But then it means that the entropy of X relative to any probability measure is negative ? – MLass Jun 03 '21 at 14:27
  • Well, if your definition of entropy relative to another probability measure is simply negative KL, then yeah. – E-A Jun 03 '21 at 14:33
  • It is little bit disturbing but OK (I guess). Another thing which is not clear for me : why the entropy of X relative to the counting measure (Shannon entropy), has all the well known properties and it is not the case using the Lebesgue measure (differential entropy). Why the counting measure is so special ? – MLass Jun 03 '21 at 14:45
  • 1
    I would not say so much as "counting measure is special" so much as in discrete land, it is sum of log of probabilities; densities are not probabilities. – E-A Jun 03 '21 at 16:00