# Mutual information between a vector and a constant

Let's say that we have a tensor x sampled from its probability distribution, and that we have a constant vector c sampled from a degenerate distribution (its value is constant).

Is the mutual information between the tensor x in R^(n, m) and the constant tensor c in R^(n, m) defined? What is its value (I imagine it should be zero)?

• It depends on the probability distribution and on the direction in which you are taking the mutual information. There's nothing special about being a "tensor" here: you might as well ask the question for a general probability space. – whuber May 26 at 12:53
• @whuber thank you for your kind reply :) Can I ask you to elaborate more on the first sentence? – gab May 26 at 14:43
• For starters, the very meaning and mathematical definition of mutual information varies with the distribution of $\mathbf x:$ it has one meaning for discrete distributions and another for continuous ones. It's possible in the discrete case for the mutual information to be finite but not in the continuous case, because the distribution of the constant is singular relative to any continuous distribution. – whuber May 26 at 15:52
• I'm not sure I understand. Could you answer adding some examples/mathematical proof, please? – gab May 27 at 10:47

The mutual information is equal to $$I(X,Y) \overset{\text{def}}{=} H(Y) - H(Y|X) = H(X) - H(X|Y) =$$ $$=\int_{\text{Supp}(X)} p(x)\log(p(x))dx - \int_{\text{Supp}(Y)} p(x|y)\log(p(x|y))dx$$ where $$x \in \text{Supp}(X) \overset{\text{def}}{\iff} p(x) \neq 0$$ and the convention is that $$0 \log 0 = 0$$ since we know that $$\lim_{z \to 0}z \log z = 0$$

Further notice that $$I(X,Y) = I(Y,X)$$, (the proof is trivial, see eq. 2.46 in Cover and Thomas).

Setting

$$X$$ = "constant vector c sampled from a degenerate distribution (its value is constant)"

and

$$Y$$ = "tensor x sampled from its probability distribution,"

we have that

$$I(X,Y) = 0$$

since $$1 \log 1 = 0 \implies \int_{\text{Supp}(X)} p(x)\log(p(x))dx - \int_{\text{Supp}(Y)} p(x|y)\log(p(x|y))dx = 0.$$