1
$\begingroup$

I know that mutual information is a measure for how similar two random variables are. There is plenty of information available about how to empirically determine mutual information empirically from two random variables that are scalars.

My question: Is there a definition for mutual information between randomly generated vectors?

Empirically, if I stored each realization of the random variable as row in a matrix with the realizations of the corresponding variable as rows in a another matrix, is there a way to calculate the mutual information between these random variables?

(Also these variables have different dimensions. One is R^2 while the other is R^6)

$\endgroup$
1
  • $\begingroup$ What do you mean by “how similar two random variables are”? I think of mutual information as being about the dependence between two variables. Do you mean something like a multivariate Kolmogorov-Smirnov test? $\endgroup$
    – Dave
    Commented Mar 23, 2023 at 4:07

2 Answers 2

1
$\begingroup$

The JMI R package implements a “jackknife mutual information” that can apply to vectors. The idea was developed by Zeng et al (2018)

The JMI::JMI function that does the calculation takes matrix inputs, just as you describe, and these matrices need not have the same number of columns (different vector lengths).

From what I recall, this implementation is slow. However, it may strike you as good news that the PNAS article works out the idea (and has references) and that there is even a software implementation!

REFERENCE

Zeng, Xianli, Yingcun Xia, and Howell Tong. "Jackknife approach to the estimation of mutual information." Proceedings of the National Academy of Sciences 115.40 (2018): 9956-9961.

$\endgroup$
1
  • $\begingroup$ This is interesting, it looks like the author's don't put an emphasis on this being a more generalized test (i.e. moving from random scalars to random vectors). Looks like the selling point is that you don't have to tune your bin widths $\endgroup$ Commented Mar 23, 2023 at 4:30
0
$\begingroup$

As far as definition is concerned, there is no problem in defining the mutual information between two (vector) random variables that take value in different spaces. The mutual information between two random variables $X \in \mathcal{X}$ and $Y \in \mathcal{Y}$ with joint distribution $p_{X,Y}$ and marginals $p_X$ and $p_Y$, respectively, is defined as $$ \begin{align} I(X;Y) &= \sum_{x \in \mathcal{X}}\sum_{y \in \mathcal{Y}} p_{X,Y}(x,y) \log \frac{p_{X,Y}(x,y)}{p_X(x)p_Y(y)}\\ &=H(X)+H(Y)-H(X,Y). \end{align} $$

If you estimate the distributions $p_{X,Y}$, $p_X$, $p_Y$, you should be able to compute a (naive?) estimation for the mutual information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.