Let's say that we have a tensor x sampled from its probability distribution, and that we have a constant vector c sampled from a degenerate distribution (its value is constant).
Is the mutual information between the tensor x in R^(n, m) and the constant tensor c in R^(n, m) defined? What is its value (I imagine it should be zero)?
The mutual information is equal to $$I(X,Y) \overset{\text{def}}{=} H(Y) - H(Y|X) = H(X) - H(X|Y) = $$ $$=\int_{\text{Supp}(X)} p(x)\log(p(x))dx - \int_{\text{Supp}(Y)} p(x|y)\log(p(x|y))dx $$ where $$ x \in \text{Supp}(X) \overset{\text{def}}{\iff} p(x) \neq 0 $$ and the convention is that $$ 0 \log 0 = 0 $$ since we know that $$ \lim_{z \to 0}z \log z = 0 $$
Further notice that $I(X,Y) = I(Y,X)$, (the proof is trivial, see eq. 2.46 in Cover and Thomas).
Setting
$X$ = "constant vector c sampled from a degenerate distribution (its value is constant)"
and
$Y$ = "tensor x sampled from its probability distribution,"
we have that
$$I(X,Y) = 0 $$
since $$ 1 \log 1 = 0 \implies \int_{\text{Supp}(X)} p(x)\log(p(x))dx - \int_{\text{Supp}(Y)} p(x|y)\log(p(x|y))dx = 0.$$
