3
$\begingroup$

To the best of my knowledge, mutual information (MI) is zero if and only if the variables are independent. I have simulated copula data and computed the MI and the results are as follows:

  1. Independent copula (MI = 0.04)
  2. Very weak dependency Gaussian copula (MI = -0.03)!

  • My question is, is it acceptable to have negative MI? Is that because the relationship between mutual information and copula is negative?

  • Is it acceptable to have > 0 for completely independent variables.

I used an 'infotheo` package in R.

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ Perhaps this is the case: You are approximating the mutual information based on finite sample. Whoever wrote that package, did not use a good estimator of the mutual information to at least produce a nonnegative result. (The mutual information is always nonnegative.) $\endgroup$
    – passerby51
    Commented Jul 8, 2021 at 6:18
  • $\begingroup$ @passerby51 Thanks for your comment and help. I tried the mm estimator (Miller-Madow asymptotic bias-corrected empirical estimator). It is the only estimator that can give me a very close value to the correct one. $\endgroup$
    – Maryam
    Commented Jul 8, 2021 at 6:22
  • 3
    $\begingroup$ If it's bias-corrected may explain why the estimate is negative when the true mutual information is zero: it's hard for a non-negative estimator of zero to be unbiased. Also, do you mean the infotheo package? $\endgroup$
    – Thomas Lumley
    Commented Jul 8, 2021 at 6:40
  • $\begingroup$ @ThomasLumley Thanks a lot for your comment and help. Yes, it is infotheo package. So, do you advise me to use another estimator? $\endgroup$
    – Maryam
    Commented Jul 8, 2021 at 6:48
  • 2
    $\begingroup$ Whatever estimator you use, say $\hat I$, I would just define a new estimator based on it say $\hat J = \max(\hat I, 0)$. This would be a strictly better estimator (It could be biased, but who cares! This is one example where you should trade-off bias for something else.) Regarding "Is it acceptable to have > 0 ..", you just have to understand the "sampling error" (the inherent uncertainly caused by looking at a finite sample of a distribution). Then it would hopefully make perfect sense. $\endgroup$
    – passerby51
    Commented Jul 8, 2021 at 7:02

1 Answer 1

Reset to default
2
$\begingroup$

The mutual information should be always non-negative. So whatever estimator you use, say $\hat I$, I would just define a new estimator based on it say $\hat J= \max(\hat I,0)$. If $\hat I$ is always nonnegative, $\hat J = \hat I$. Otherwise $\hat J$ would be a strictly better estimator (closer to the truth). $\hat J$ could be biased, but who cares! This is one example where you should trade-off bias for something else.

Personally, I would use an estimator based on the KL divergence definition of the mutual information which is guaranteed to be nnonegative, not based on estimating the entropies in $ H(X) + H(Y) - H(X,Y)$. The idea is to use binning, compute a contingency table (a discretized version of the joint distribution), and then compute the KL divergence between the joint distribution and the distribution with corresponding independent marginals. (There should be ways to fix the bias you get due to binning if the number of samples in each bin is very small. Otherwise this should be a good estimate in itself. Choosing the bin size is generally the tricky problem).

Regarding "Is it acceptable to have > 0 for completely independent variables", you just have to understand the "sampling error" (the inherent uncertainly caused by looking at a finite sample of a distribution). Then it would hopefully make perfect sense.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.