3
$\begingroup$

How does one compute mutual information (MI) between discrete random variables and continuous random variables ?

It's easy to compute MI for dataset with only discrete random variables. For dataset with continuous random variables, MI can be computed by discretising the continuous random variable.

However, when the dataset contains both discrete and continuous random variables, how do I compute the MI between these two types of random variables ?

Any credible recommended papers?

$\endgroup$
1
  • $\begingroup$ MI is defined for continuous random variables without discretizing them first. And, if discretizing is a sufficient approximation when dealing exclusively with continuous variables, why not use discretization for the mixed situation, too? $\endgroup$
    – frank
    Commented Jul 19, 2022 at 10:45

1 Answer 1

2
$\begingroup$

Note that there is a difference between defining and calculating the MI of two random variables $X$, $Y$ whose joint distribution you know and actually estimating the MI from data $\left(X_i, Y_i\right)_{i = 1, \dots, n}$.

In any case see e.g. [1] for an estimator of MI for continuous variables without discretization, [2] and [3] for extensions of this estimator to discrete-continuous settings. [3] also has Definition 2.1 where MI is defined in terms of the Radon-Nikodym derivative of the joint distribution $\mathbf P^{X,Y}$ w.r.t. the product measure $\mathbf P^X \otimes \mathbf P^X$ which is valid for all types of random variables (essentially MI is the KL-divergence between these two measures). [3] also gives some guarantees, e.g. that the estimated MI is consistent for $n \to \infty$ (under some conditions). Off the top of my hat I unfortunately do not know about more rigorous papers (though I don't doubt that they exist).

[1]A. Kraskov, H. Stoegbauer, and P. Grassberger, ‘Estimating Mutual Information’, Phys. Rev. E, vol. 69, no. 6, p. 066138, Jun. 2004, doi: 10.1103/PhysRevE.69.066138.

[2]B. C. Ross, ‘Mutual Information between Discrete and Continuous Data Sets’, PLoS ONE, vol. 9, no. 2, p. e87357, Feb. 2014, doi: 10.1371/journal.pone.0087357.

[3]W. Gao, S. Kannan, S. Oh, and P. Viswanath, ‘Estimating Mutual Information for Discrete-Continuous Mixtures’, in Advances in Neural Information Processing Systems, 2017, vol. 30. Accessed: May 24, 2022. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/ef72d53990bc4805684c9b61fa64a102-Abstract.html

$\endgroup$
1
  • 2
    $\begingroup$ is there any python packages /code that implement the methodologies stated in these 3 papers ? $\endgroup$ Commented Jul 20, 2022 at 2:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.