2

I am reading this paper Kraskov et al, 2004, Estimating Mutual Information on estimating the mutual information of two continuous random variables based on entropy estimates from $k$-nearest neighbour distances. I do not quite understand the motivation is using $k$-nearest neighbour distances. Could someone elucidate it?

This paper is also referred to in this answer.

Hans
  • 10,484

1 Answers1

3

Here's the intuition from my perspective.

First at a high level, the mutual information (MI) is measuring the (expected, logarithmic) distance between the joint distribution $p(x,y)$ and the product of the marginals $p(x)p(y)$. This means you need some kind of way to estimate densities. The obvious way is to make a histogram, and count the (normalized) number of points falling into each bin.

However, keep in mind that we are ultimately interested in the probability density, which is directly related to the sample density. But, just intuitively, samples are dense in a particular location if there are many of them close together; i.e., the distances to their nearest neighbours are small. In other words, areas of high density have densely packed samples, meaning the distance between nearby points is small. Conversely, areas of low density have few samples, and so the distance to their closest neighbours will be large.

Hence, the paper tries to

estimate MI from k-nearest neighbour statistics

So basically we can just think of using neighbour distances as another proxy for sample density, instead of counting samples in bins. It sort of reminds me of kernel density estimation.

More concretely, for this paper, however, they are thinking more about a different perspective on the mutual information, namely as the difference between the independent (marginal) entropies and the joint entropy. They then use estimators for the entropies, based on the neighbour statistics. Another intuition, I think, is that counting samples based on local neighbourhood distances (in the joint space) makes the algorithm adaptive (as opposed to choosing a single bin size over the whole sample space).

user3658307
  • 10,843
  • What is "it" referring to in "It sort of reminds me of kernel density estimation", the $k$-nearest neighbour or counting samples in bins? To me, the kernel density estimation is akin to the histogram only that this histogram is overlapping and "soft". What puzzles is then which $k$ one should choose? Should we sum or average over all $k$'s or leave it as a hyperparameter to be determined by, say, cross-validation? Also, which is better, $k$-nearest neighbour or kenel density estimation? – Hans Jan 12 '19 at 07:40
  • By the way, you may be interested in and have an opinion on my stats.stackexchange.com question https://stats.stackexchange.com/q/386101/44368. – Hans Jan 12 '19 at 07:45
  • @Hans $k$ is chosen as a fixed value, I believe. Some discussion of the tradeoff of higher vs lower $k$ is talked about in the paper (e.g., computational cost, relation to the scale of the data). You have to choose $k$ based on these considerations. KDE is indeed like a soft histogram. "It" was just referring to the nearest neighbour method. KDE also has several (difficult) parameters to choose: e.g., which kernel? which bandwidth? I am not an expert in this area, but I suspect that both methods can be accurate in practice, but it depends on the chosen parameters and the data :) – user3658307 Jan 12 '19 at 14:52