16
$\begingroup$

I have a very basic doubt. Sorry if this irritates few. I know that Mutual Information value should be greater than 0, but should it be less than 1 ? Is it bounded by any upper value ?

Thanks, Amit.

$\endgroup$

2 Answers 2

26
$\begingroup$

Yes, it does have an upper bound, but not 1.

The mutual information (in bits) is 1 when two parties (statistically) share one bit of information. However, they can share a arbitrary large data. In particular, if they share 2 bits, then it is 2.

The mutual information is bounded from above by the Shannon entropy of probability distributions for single parties, i.e. $I(X,Y) \leq \min \left[ H(X), H(Y) \right]$ .

$\endgroup$
1
  • $\begingroup$ If the two parties $X,Y$ are binary variables i.e. each has only two possible outcomes {0,1}, then entropies $H(X), H(Y)$ max out at $1$ when $P(X)=0.5$ and $P(Y)=0.5$. Thus, maximum mutual information for two binary variables is $1$ $\endgroup$ Jul 4, 2019 at 14:43
1
$\begingroup$

It depends on whether the alphabet of interest is finite with a known finite cardinality $K$, a finite but unknown cardinality $K$, or an infinite countable alphabet. If you are talking about mutual information (there is a confusion in names, for example, mutual information, information gain, information gain ratio, etc.), then the answer is YES if $K$ is known, NO if$K$ is unknown or infinite - mutual information is unbounded on a countable alphabet!

The answer provided above is incorrect because $I(X,Y)\leq \min(H(X),H(Y))$ is an incorrect statement. This is easily seen since $H(X)$ and $H(Y)$ may be arbitrarily large. That said however it must be mentioned that the author of the answer above may be thinking $K$ being a known integer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.