Let $X$ and $Y$ be two random variables, where $Y=f(X)$ is a deterministic function of $X$. Furthermore suppose $X,Y$ are continuous and that $f$ is smooth.
Is the mutual information between $X$ and $Y$ well defined in this case? I have the intuition that it can only be zero (if $f$ is a constant function) or infinity.
Let's suppose that $Y=f(X)+\epsilon Z$ where $Z$ is a standard normal, and we take the limit $\epsilon\rightarrow 0$
The mutual information equals $$I = H(Y) - H(Y|X),$$ where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the conditional entropy of $Y$ on $X$. Since $Z$ is normal, $H(Y|X)=\frac{1}{2}\ln(2\pi e\epsilon^2)$.
If $f$ is not constant and smooth, then $H(Y)$ stays positive and finite. But as $\epsilon\rightarrow 0$ we find $H(Y|X)\rightarrow-\infty$ and therefore $I=\infty$ in this limit.
If $f(X)=c$ is a constant, then $H(Y)=H(Y|X)$ for any positive $\epsilon$, and therefore in this case the limit is $I=0$.
Although $f$ is stated as neural network, I think theorem 1 of paper https://arxiv.org/pdf/1802.09766.pdf will answer part of your question, i.e. , if $f$ is bi-Lipschitz then $I(Y, X) = \infty$.
Furthermore, $f(X) = c$ is not a bi-Lipschitz function. A example of piecewise linear function， a general form of constant function， is present in section 4.3； I think at least for that example the mutual information is a finite constant for piecewise linear function.
