1
$\begingroup$

This question already has an answer here:

library(infotheo)
> mutinformation(c(1,2,3),c(1,2,3))
[1] 1.098612
> mutinformation(c(1,1,1),c(1,1,1))
[1] 0

Why is there a difference?

I keep reading that this should either be 0 or infinity, yet I get neither

Is this because I have not defined the joint probability?

Why would perfectly similar data have 0 mutual information?

$\endgroup$

marked as duplicate by Tim, Michael Chernick, kjetil b halvorsen, mdewey, mpiktas Jun 23 '17 at 8:06

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ The mutual information coefficient measures how related two variables are, but it has a lot to do with their joint and separate entropy. Two variables that are constant have zero entropy, hence zero mutual information. en.wikipedia.org/wiki/Mutual_information $\endgroup$ – Anna SdTC Mar 14 '17 at 1:49
1
$\begingroup$

Entropy and therefore mutual information depend on the probability distribution of your data, not on the values your data takes.

In your case the you have the vectors:

V1 = {1,2,3} and V2 = {1,1,1}

In the first case, the probability of each element is 1/3, in the second case, the probability of each element is 1:

p(V1) = {1/3,1/3,1/3} and p(V2) = {1}

This means that your first vector (V1) has an entropy(total information) of 1.584 bits. You need on average 1.584 bits to describe your vector.

In the case of your second vector, there is no information. You know it always contains a 1. Hence it's entropy equals 0.

When you compute the mutual information between a random variable and itself, you always end up with the total information of your random variable:

I(X,X) = H(X) 
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.