I'm thinking if there's a way of interpreting mean square error loss frequently applied in statistics and machine learning under the information framework?

Suppose we have two random variables (or vectors), $X$ and $Y$, and let's denote their observations by $\tilde{X}_i$ and $\tilde{Y}_i$, $i=1,\ldots,n$. Our job is to predict $Y$ as best as we can. In regression tasks, one is often encouraged to find a function $f$ such that $L(f) = \frac{1}{m}\sum_i (f(\tilde{X_i})-\tilde{Y_i})^2$, the squared error, is minimized. We often feel comfortable to apply $f$ on $X$ to predict $Y$.

From an intuitive point of view, if we denote all possible regressors are in the set $\Omega$, and if $\min_{f\in\Omega}L(f)$ is sufficiently, we can say that $X$ contains enough "information" about $Y$. Now the function $f$ minimizing the loss globally may not be found, but once you find a $f$ such that $L(f)$ is obtained, I suppose, in principle, one can obtain a lower bound of how much information $X$ contains about $Y$.

In information theory, the idea of information is formalized by the self-information w.r.t a random variable, $H(X), H(Y)$, mutual information between two variables, $I(X,Y) = H(X) - H(X|Y)$ and the K-L divergence $D(Y||X)$. My question is, can we find a lower bound for $I(Y|X)$ or $D(Y||X)$ once we are given a regressor $f$ and its MSE $L(f)$, without specifying the distribution of $Y$ or $X$? Or is it in principle possible?

Another often used loss function is the cross entropy loss. Would the problem become easier if we replace MSE with cross entropy loss, since it is in form consistence with the definition of information?

  • $\begingroup$ There is the 'no-free-lunch' theorem stating that the error a concrete predictor $f$ makes must be 'big in average', i.e. for a given learning strategy we can find a distribution $p(X,Y)$ such that the learning strategy fails miserably (is arbitrarily close to guessing). In general Im unsure about the nature of the question: I(X,Y) is something that does not involve neither $f$ nor tha strategy how to build $f$... so, what do you mean? How big/small I(X,Y) can be if we do not know anything about X and Y? Well... it can be 0 if they are independent... $\endgroup$ Commented Nov 2, 2017 at 16:33
  • $\begingroup$ @FabianWerner Thanks for the comment. I mean if we find a f such that the mse loss is L(f), we know that the L(f) is an upper bound of the minimal error we can get from a predictor, since ff achieves it. I guess less loss means more information about Y contained in X, so is it possible to derive a lower bound for $I(X,Y)$, in this case? $\endgroup$
    – R. Feng
    Commented Nov 2, 2017 at 16:47
  • $\begingroup$ @FabianWerner I realize that my question can be ill-formed or primitive. Maybe without constraint on $f$ or how it is built, the question is meaningless. $\endgroup$
    – R. Feng
    Commented Nov 2, 2017 at 16:48
  • $\begingroup$ Well... yes, there is a lower bound: It is always $\geq 0$ and it can be exactly zero if $X$ is completely unrelated to $Y$... so, (just trying to figure out the 'right question') isn't it a lower bound on the performance/error of a predictor that you are actually after? Or is it a relation like 'if I(X,Y) is that small then we can find no better predictor than ...'? For example: If I(X,Y)=0 then we can only guess and we will never do better than guessing... $\endgroup$ Commented Nov 2, 2017 at 17:02
  • $\begingroup$ @FabianWerner In that case, given the observations of X and Y, can I estimate H(X), H(Y), and I(X, Y)? $\endgroup$
    – R. Feng
    Commented Nov 3, 2017 at 14:45

1 Answer 1


If we can predict X from Y with some accuracy, this in general gives us a lower bound on mutual information. Our predictor certifies that X has a certain amount of information about Y, but we only have a lower bound because it could be that a different predictor would work better. In the discrete case, this is summarized with Fano's inequality (keep in mind $I(X,Y) = H(X)-H(X|Y)$ when looking at this page), which shows how mutual information is bounded by prediction error.

For the continuous case, it is possible to derive similar bounds. One example would be the following: \begin{align} I(X,Y) &= H(Y)-H(Y|X) = H(Y) + \mathbb E_p \log p(y|x) \\ &\geq H(Y) + \mathbb E_p \log q(y|x) \\ &= H(Y) - 1/2 \log \mathbb E_p[(Y-f(X))^2] -1/2\log (2 \pi e) \end{align} In the second line, we use the non-negativity of KL divergence, introducing some variational distribution, $q$. In the third line, we make a specific choice for $q(y|x) \sim \mathcal N(f(x), E_p[(Y-f(X))^2])$. As you can see, the smaller the error in your regression, the larger the lower bound on mutual information will become. Through different choices of the variational distribution, you can get different bounds. (In case it is not clear, $\mathbb E_p$ is the expectation and the sample expectation would just be $\mathbb E_p[ (Y-f(X))^2] = 1/m \sum_{i=1}^m (Y_i-f(X_i))^2$.)

  • $\begingroup$ is this trick to obtain a "continuous Fano inequality" from somewhere? Could you suggest a reference? $\endgroup$
    – glS
    Commented Jun 12, 2023 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.