I need to compute mutual information gain based two continuous variables $X$ and $Y$

$I(X|Y) = \int_X\int_Y p_{x.y}(x,y) \log(\frac{p_{x.y}(x,y)}{p_{x}(x)p_{y}(y)})$.

I have used Kernel Density Estimation (KDE) for generating 2D density $f_{x.y}(x,y)$. So I have matrix $m \times n$.

The question is, how should I choose parameter $m$ and $n$. Generating 2d probability density using KDE is very time consuming so I have to find minimal $m$ and $n$ but I don't want to lose a lot of information. Is there any general rule which can ensure that the probability density is well sampled?

  • $\begingroup$ There is a rule of thumb that is often used with binned estimators of MI (i.e. estimators in which the probabilities are replaced with histograms). It states that the number of samples should be approximately 5 times the number of bins chosen to discretize the $p(x,y)$. I don't remember where the rule originated (I think from the bias-correction theories), but it is used in practice. Maybe it is useful for your case as well. Also, beware that your estimates will be biased. $\endgroup$ – Cesare Aug 26 '19 at 8:38
  • $\begingroup$ Thank you very much. I will try to find more information about this. :) $\endgroup$ – Jarryh Aug 27 '19 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.