11
$\begingroup$

I have a very basic doubt. Sorry if this irritates few. I know that Mutual Information value should be greater than 0, but should it be less than 1 ? Is it bounded by any upper value ?

Thanks, Amit.

$\endgroup$
18
$\begingroup$

Yes, it does have an upper bound, but not 1.

The mutual information (in bits) is 1 when two parties (statistically) share one bit of information. However, they can share a arbitrary large data. In particular, if they share 2 bits, then it is 2.

The mutual information is bounded from above by the Shannon entropy of probability distributions for single parties, i.e. $I(X,Y) \leq \min \left[ H(X), H(Y) \right]$ .

$\endgroup$
  • $\begingroup$ If the two parties $X,Y$ are binary variables i.e. each has only two possible outcomes {0,1}, then entropies $H(X), H(Y)$ max out at $1$ when $P(X)=0.5$ and $P(Y)=0.5$. Thus, maximum mutual information for two binary variables is $1$ $\endgroup$ – Akseli Palén Jul 4 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.