Is mutual information invariant to scaling, i.e. multiplying all elements by a nonzero constant?

Question

I am trying to compare it to Euclidean distance and Pearson correlation

Answer 1

More generally, mutual information is invariant under any smooth and uniquely invertible transformation of the variables.

See "Estimating mutual information" by A Kraskov, H Stögbauer, P Grassberger - Physical Review E, 2004 [http://arxiv.org/pdf/cond-mat/0305641]

Answer 2

I think the answer is yes to your question. I will show this for the discrete case only and I think the basic idea carries over to the continuous case. MI is defined as:

$I(X;Y) = \sum_{y\in Y}\sum_{x\in X}\Bigg(p(x,y) log(\frac{p(x,y)}{p(x)p(y)})\Bigg)$

Define:

$Z_x = \alpha X$

and

$Z_y = \alpha Y$.

So, the question is: Does $I(Z_x;Z_y)$ equal $I(X;Y)$?

Since scaling is a one-to-one transformation it must be that:

$p(z_x) = p(x)$,

$p(z_y) = p(y)$ and

$p(z_x,z_y) = p(x,y)$

Therefore, the mutual information remains the same and hence the answer is to your question is yes.

Answer 3

Intuitive explanation is such: multiplying by constant does not change information content of X and Y, so also their mutual information -- and thus it is invariant to scaling. Still Srikant gave you a strict proof of this fact.