# mutual information between discrete random variables and continuous variables

How does one compute mutual information (MI) between discrete random variables and continuous random variables ?

It's easy to compute MI for dataset with only discrete random variables. For dataset with continuous random variables, MI can be computed by discretising the continuous random variable.

However, when the dataset contains both discrete and continuous random variables, how do I compute the MI between these two types of random variables ?

Any credible recommended papers?

• MI is defined for continuous random variables without discretizing them first. And, if discretizing is a sufficient approximation when dealing exclusively with continuous variables, why not use discretization for the mixed situation, too? Commented Jul 19, 2022 at 10:45

Note that there is a difference between defining and calculating the MI of two random variables $$X$$, $$Y$$ whose joint distribution you know and actually estimating the MI from data $$\left(X_i, Y_i\right)_{i = 1, \dots, n}$$.
In any case see e.g. [1] for an estimator of MI for continuous variables without discretization, [2] and [3] for extensions of this estimator to discrete-continuous settings. [3] also has Definition 2.1 where MI is defined in terms of the Radon-Nikodym derivative of the joint distribution $$\mathbf P^{X,Y}$$ w.r.t. the product measure $$\mathbf P^X \otimes \mathbf P^X$$ which is valid for all types of random variables (essentially MI is the KL-divergence between these two measures). [3] also gives some guarantees, e.g. that the estimated MI is consistent for $$n \to \infty$$ (under some conditions). Off the top of my hat I unfortunately do not know about more rigorous papers (though I don't doubt that they exist).