5
$\begingroup$

The standard mutual information (MI) is given by $$I(X;Y) = H(X) + H(Y) - H(XY)$$ which is the amount of information shared by the two random variables $X$ and $Y$.

According to wikipedia article, the multivariate mutual information (MMI) in 3D case is defined as $$I(X;Y;Z) = H(X) + H(Y) + H(Z) - H(XY) - H(XZ) - H(YZ) + H(XYZ)$$ which is the gray core of the 3-circle Venn diagram in the article.

I am interested in a related metric I derived using naive extrapolation of the 2D case. I will call it $J$ since I don't know how it is called or if it has a name.

$$J(X;Y;Z)= H(X) + H(Y) + H(Z) - H(XYZ)$$

My metric $J$ would cover the whole core, excluding only the information that is unique to each variable alone. This metric makes sense to me, because it is non-negative and is only zero if all variables are independent.

Questions:

  • Does $J$ already have a name in the literature?
  • I have some multidimensional data. My null hypothesis is that all variables are completely independent. Is $J > \epsilon(N)$ a good test for this purpose? Under null hypothesis $J = 0$, but I assume it should be compared to some correction $\epsilon(N)$ due to finite data size $N$
  • I notice that $J(X;Y;Z)$ double-counts the core $I(X;Y;Z)$. Thus, perhaps it is meaningful to define the variable $K(X;Y;Z) = J(X;Y;Z)-I(X;Y;Z)$. Does this variable have a name?
Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ As for your last question - the closest notion to the one you propose is the dual-total-correlation (due to Han). It is a close definition to the TC but in a way is doesn’t suffer from the double counting you mentioned. See en.wikipedia.org/wiki/Dual_total_correlation. $\endgroup$
    – Meni
    Commented Oct 3, 2019 at 12:02
  • $\begingroup$ the first form of the 3D case you described, MMI, or $I(X;Y;Z)$, looks like a scalar that encapsulates all the variables simultaneously. Does that make it different than an MI matrix (an approach not listed) whose elements are instead pair-wise $I(X;Y)$'s? $\endgroup$
    – develarist
    Commented Aug 14, 2020 at 18:12
  • 1
    $\begingroup$ @develarist yes, there is a fundamental difference between pairwise interactions and higher-dimensional interactions. A classical example of a triplet interaction is the XOR operation C = A XOR B. Given any two of the three variables, the last one can be computed by XOR of the first two. So, given all 3 of these variables, it can be concluded that any one of them is redundant, but that knowledge cannot be gained from inspecting any pair of the numbers. This effect is called en.wikipedia.org/wiki/Redundancy_(information_theory). It is far from the only higher dimensional effect. $\endgroup$
    – Aleksejs Fomins
    Commented Aug 14, 2020 at 21:29
  • $\begingroup$ so the MI matrix made up of individual $I(X;Y)$s which captures pair-wise interactions is a naive estimator compared to the MMI scalar $I(X;Y;Z)$ which captures higher-dimensional interactions? $\endgroup$
    – develarist
    Commented Aug 14, 2020 at 22:02
  • $\begingroup$ @develarist I'm not sure what you mean by naive. $I(X;Y)$ and $I(X;Y;Z)$ are two different estimators, they measure different things. Neither of them is better or worse than the other. Either of them can be useful, depending on what question you are interested in $\endgroup$
    – Aleksejs Fomins
    Commented Aug 17, 2020 at 7:24

1 Answer 1

Reset to default
2
+50
$\begingroup$

The functional you refer to as $J$ is called the total correlation, proposed by Watanabe (1960). It has also been called multiinformation (e.g. see Studeny and Vejnarova 1998).

Regarding independence testing, it might make sense to perform a permutation test using total correlation (TC) as the test statistic. Separately permuting the observed values of each variable preserves the empirical marginal distributions, but destroys any dependence. You can sample from the null distribution of TC (i.e. assuming independence) by repeatedly generating permuted datasets and estimating the TC for each. This can be used to calculate a p value for the TC estimated from the actual data.

But, the difficulty of estimating entropy in the multivariate case might be a concern. For some relevant citations, see this post. So, I don't know whether a permutation test based on TC would work well (this also depends on the data). At the very least, care in estimating the entropy would be needed. As a possible alternative, nonparametric tests for dependence between multiple variables have been proposed in the literature.

References:

  • Watanabe (1960). Information theoretical analysis of multivariate correlation.
  • Studeny and Vejnarova (1998). The multiinformation function as a tool for measuring stochastic dependence.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks a lot for your answer. It does not fully answer my question, but it gives me a way to find it myself, which is good enough. I appreciate the linked papers, I will read them. I am well aware that entropy estimation is a nightmare, I am using a few libraries that have super-sophisticated ways of achieving estimator stability and bias-variance compromise. $\endgroup$
    – Aleksejs Fomins
    Commented Feb 11, 2019 at 12:50
  • 1
    $\begingroup$ @AleksejsFomins Glad to hear you're covered on the entropy estimation front. Yes, it's not a complete answer, and hopefully someone will post more. But, I'm glad it's enough to help get started. Good luck $\endgroup$
    – user20160
    Commented Feb 11, 2019 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.