Let's say that we have a tensor x sampled from its probability distribution, and that we have a constant vector c sampled from a degenerate distribution (its value is constant).

Is the mutual information between the tensor x in R^(n, m) and the constant tensor c in R^(n, m) defined? What is its value (I imagine it should be zero)?

  • 1
    $\begingroup$ It depends on the probability distribution and on the direction in which you are taking the mutual information. There's nothing special about being a "tensor" here: you might as well ask the question for a general probability space. $\endgroup$ – whuber May 26 at 12:53
  • $\begingroup$ @whuber thank you for your kind reply :) Can I ask you to elaborate more on the first sentence? $\endgroup$ – gab May 26 at 14:43
  • $\begingroup$ For starters, the very meaning and mathematical definition of mutual information varies with the distribution of $\mathbf x:$ it has one meaning for discrete distributions and another for continuous ones. It's possible in the discrete case for the mutual information to be finite but not in the continuous case, because the distribution of the constant is singular relative to any continuous distribution. $\endgroup$ – whuber May 26 at 15:52
  • $\begingroup$ I'm not sure I understand. Could you answer adding some examples/mathematical proof, please? $\endgroup$ – gab May 27 at 10:47

The mutual information is equal to $$I(X,Y) \overset{\text{def}}{=} H(Y) - H(Y|X) = H(X) - H(X|Y) = $$ $$=\int_{\text{Supp}(X)} p(x)\log(p(x))dx - \int_{\text{Supp}(Y)} p(x|y)\log(p(x|y))dx $$ where $$ x \in \text{Supp}(X) \overset{\text{def}}{\iff} p(x) \neq 0 $$ and the convention is that $$ 0 \log 0 = 0 $$ since we know that $$ \lim_{z \to 0}z \log z = 0 $$

Further notice that $I(X,Y) = I(Y,X)$, (the proof is trivial, see eq. 2.46 in Cover and Thomas).


$X$ = "constant vector c sampled from a degenerate distribution (its value is constant)"


$Y$ = "tensor x sampled from its probability distribution,"

we have that

$$I(X,Y) = 0 $$

since $$ 1 \log 1 = 0 \implies \int_{\text{Supp}(X)} p(x)\log(p(x))dx - \int_{\text{Supp}(Y)} p(x|y)\log(p(x|y))dx = 0.$$

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.