MSE loss and information I'm thinking if there's a way of interpreting mean square error loss frequently applied in statistics and machine learning under the information framework?
Suppose we have two random variables (or vectors), $X$ and $Y$, and let's denote their observations by $\tilde{X}_i$ and $\tilde{Y}_i$, $i=1,\ldots,n$. Our job is to predict $Y$ as best as we can. In regression tasks, one is often encouraged to find a function $f$ such that $L(f) = \frac{1}{m}\sum_i (f(\tilde{X_i})-\tilde{Y_i})^2$, the squared error, is minimized. We often feel comfortable to apply $f$ on $X$ to predict $Y$. 
From an intuitive point of view, if we denote all possible regressors are in the set $\Omega$, and if $\min_{f\in\Omega}L(f)$ is sufficiently, we can say that $X$ contains enough "information" about $Y$. Now the function $f$ minimizing the loss globally may not be found, but once you find a $f$ such that $L(f)$ is obtained, I suppose, in principle, one can obtain a lower bound of how much information $X$ contains about $Y$. 
In information theory, the idea of information is formalized by the self-information w.r.t a random variable, $H(X), H(Y)$, mutual information between two variables, $I(X,Y) = H(X) - H(X|Y)$ and the K-L divergence $D(Y||X)$. My question is, can we find a lower bound for $I(Y|X)$ or $D(Y||X)$ once we are given a regressor $f$ and its MSE $L(f)$, without specifying the distribution of $Y$ or $X$? Or is it in principle possible? 
Another often used loss function is the cross entropy loss. Would the problem become easier if we replace MSE with cross entropy loss, since it is in form consistence with the definition of information? 
 A: If we can predict X from Y with some accuracy, this in general gives us a lower bound on mutual information. Our predictor certifies that X has a certain amount of information about Y, but we only have a lower bound because it could be that a different predictor would work better. In the discrete case, this is summarized with Fano's inequality (keep in mind $I(X,Y) = H(X)-H(X|Y)$ when looking at this page), which shows how mutual information is bounded by prediction error. 
For the continuous case, it is possible to derive similar bounds. One example would be the following: 
\begin{align}
I(X,Y) &= H(Y)-H(Y|X) = H(Y) + \mathbb E_p \log p(y|x) \\
&\geq  H(Y) + \mathbb E_p \log q(y|x) \\
&= H(Y) - 1/2 \log \mathbb E_p[(Y-f(X))^2] -1/2\log (2 \pi e)
\end{align}
In the second line, we use the non-negativity of KL divergence, introducing some variational distribution, $q$. In the third line, we make a specific choice for $q(y|x) \sim \mathcal N(f(x), E_p[(Y-f(X))^2])$. As you can see, the smaller the error in your regression, the larger the lower bound on mutual information will become. Through different choices of the variational distribution, you can get different bounds. (In case it is not clear, $\mathbb E_p$ is the expectation and the sample expectation would just be $\mathbb E_p[ (Y-f(X))^2] = 1/m \sum_{i=1}^m (Y_i-f(X_i))^2$.)
