5
$\begingroup$

I'm trying to wrap my head around the connection between statistical regression and its probability theoretical justification. In many books on statistics/machine learning, one is introduced to the idea of the loss function, which is then typically followed by a phrase of the flavour 'a popular choice for this function is mean squared loss'. As far as I understand, the justification for this choice stems from the theorem that

$$ \arg\min_{Z \in L^2(\mathcal{G})} \ \mathbb{E} \left[ (X - Z)^2 \right] = \mathbb{E} \left[ X \Vert \mathcal{G} \right] \tag{1} $$

where $X$ is the random variable to be estimated based on the information contained in $\mathcal{G}$. As far as I understand, probability theory teaches us that the conditional expectation $\mathbb{E}[X \Vert \mathcal{G}]$ is the best such estimate. If that's the case, why should our loss function still be a choice? Clearly we should be statistically estimating $\mathbb{E}[X \Vert \mathcal{G}]$, which by (*) implies minimizing the MSE.

An answer which I have often read is that we simply define the conditional expectation to satisfy (1), but that's doesn't seem true, as we have conditional expectations for any random variable in $L^1$. More importantly, there exists an intuitive theoretical explanation for why this definition gives us an estimator capturing all the information available after observing $\mathcal{G}$: we're using $\mathcal{G}$ to partition the total probability into possible paths and averaging over the remaining randomness in each of these. This interpretation in terms of information and $\sigma$-algebras has nothing to do, as far as I can tell, with minimizing MSE, we could have come up with it without ever knowing (1).

So my question really is: does minimizing MSE represent the theoretically optimal criterion, and if so, are we saying that any alternative (such as LAD) inherently represents a loss of theoretical optimality in favour of good estimation properties, etc.? Are necessarily leaving (as the explanation in the previous paragraph suggests) information contained in $\mathcal{G}$ on the table? And how do we quantify 'how much information' of $\mathcal{G}$ an estimator based on a different criterion (say, the median in case of LAD) utilises?

I've asked this question already on Mathematics Stack Exchange but I'm still not completely satisfied, so I was hoping someone here could maybe illuminate me. Judging by the number of similar questions on this subject, this is probably a phase all students of statistics pass through.

$\endgroup$
11
  • 1
    $\begingroup$ conditional expectation 𝔼[𝑋‖] is the best such estimate in the least squares sense. Mean absolute deviation might be appropriate in presence of outliers, in general maximising likelihood will give you lots of different answers depending on the noise distribution you assume. Start with a textbook. Do you have one? if so point to your misunderstandings. If you are learning ML then elements of statistical learning is a suitable mathematically advanced one. $\endgroup$
    – seanv507
    Commented Jan 2, 2022 at 12:49
  • 1
    $\begingroup$ "f you read arguments about measurability etc. then E[X∥G] is not just the least squares sense best estimate, but the estimate which maximally utilises the information in G by averaging over the remaining randomness" What does utilises mean? The answer depends on your goal. Some people have their own custom function like - if I predict answers in the range of (60-80) wrong that is terribly costly. I do not understand your question because the whole concept of "utilizes" or optimilaity is subjective. $\endgroup$ Commented Jan 2, 2022 at 13:41
  • 2
    $\begingroup$ Have a look at maximum likelihood estimation. $\endgroup$ Commented Jan 2, 2022 at 13:42
  • 1
    $\begingroup$ @othi yes, I think this interpretation is correct $\endgroup$ Commented Jan 2, 2022 at 13:54
  • 1
    $\begingroup$ If you use a word like all information then that too requires a definition: Fisher's variance of the score has an underlying squared term $\endgroup$
    – Henry
    Commented Jan 2, 2022 at 14:13

2 Answers 2

3
$\begingroup$

Depending on the situation, minimizing square loss might not even be the best way to estimate the mean. For instance, if data are Laplace-distributed, minimizing absolute loss gives an estimator that is, in many regards, better than minimizing square loss. Both are unbiased, but the minimization of absolute loss results in lower variance. This can be seen in a simulation.

library(ggplot2)
library(VGAM)
library(quantreg)
set.seed(2023)
N <- 10
R <- 5000
mse_based <- mae_based <- rep(NA, R)
for (i in 1:R){
  
  y <- VGAM::rlaplace(N, 0, 1)
  
  L <- lm(y ~ 1)
  Q <- quantreg::rq(y ~ 1, tau = 0.5)
  
  mse_based[i] <- summary(L)$coef[1, 1]
  mae_based[i] <- summary(Q)$coef[1]
  
  if (i %% 100 == 0){
    print(paste(
      i/R*100, 
      "% complete", 
      sep = ""
    ))
  }
  
}

d_mse <- data.frame(
  Estimate = mse_based,
  Estimation = "MSE Minimization"
)
d_mae <- data.frame(
  Estimate = mae_based,
  Estimation = "MAE Minimization"
)
d <- rbind(d_mse, d_mae)
ggplot(d, aes(x = Estimate, fill = Estimation)) +
  geom_density(alpha = 0.2) +
  theme(legend.position="bottom")

KDE of estimates

Visually, both have means around zero, which is the true value, but the MSE minimization results in considerably more variability in the estimates. This is not just a fluke of this simulation but can be shown mathematically for a Laplace distribution.

This is just one situation where an alternative to MSE minimization is theoretically justified.

Additionally, I demonstrate here that minimizing MSE need not produce a better out-of-sample MSE than minimizing MAE. This could be another justification for minimizing MAE instead of MSE.

Finally, in "classification" problems like logistic regression, making the (reasonable) assumption of a Bernoulli conditional distribution means that minimizing the log-loss, not the square loss, corresponds with maximum likelihood estimation, giving another theoretical justification for minimizing a loss function other than square loss.

$\endgroup$
1
  • 1
    $\begingroup$ And that is without even getting into regularization... $\endgroup$
    – Dave
    Commented Mar 14, 2023 at 19:25
0
$\begingroup$

When dealing with (binary) classification problems, e.g. problems where we are trying to map an input $x\in\mathcal X$ to a label $y\in\{+1;1\}$, it is easy to prove that the optimal classification rule (i.e. the classifier which minimizes the 0-1 loss) is given by $$\eta(x) = \text{sign}\ \mathbb E[Y\mid X = x] $$ Where the pair $(X,Y)$ follows the data distribution.

It thus makes sense to try to minimize the MSE $\mathbb E[(Y-f(X))^2] $, since as you said, the approximate minimizer $\hat f_{MSE}$ will be an approximation of $\mathbb E[Y\mid X] $ and therefore $\hat\eta_{MSE}:=\text{sign}\ \hat f_{MSE} $ should be a good approximation of the optimal classifier $\eta$ as well.

What we know, however, is that there exists a large family of surrogate losses $\ell$, such that any minimizer $f_\ell$ of $\mathbb E[\ell(f(X),Y)]$ satisfies $$\text{sign}f_{\ell} = \eta $$ Such loss functions are called Bayes consistent or classification calibrated, and there is an easy characterization for them and many of the practically used losses have this property (you can have a look here for more details). Concretely, this means that minimizing the loss associated to such an $\ell$ will solve our problem just as well as minimising the MSE. Hence, if the optimization problem associated with $\ell$ is easier to solve or has other desirable properties (regularization, robustness...) it makes sense to chose that option over minimizing the MSE.


The above discussion is mainly about (binary) classification, but even for that problem, some popular loss functions used in practice, such as cross-entropy, do not come with the theoretical guarantee mentioned above. Nevertheless, it happens to be very useful and exhibit many desirable properties (nicer loss landscape, stability during training...) and it arises naturally by considering the problem through another angle (look at the wikipedia page for more details).

Similarly, for regression, we may sometime prefer to minimize other metrics, such as the MAE or the MAPE rather than MSE, whose minimizers we know won't converge to the "optimal predictor", simply because we prefer our estimators to be more robust, depending on what we know about our data, or because we know that it will be more efficient computationally.

You can have a look at these threads : 1, 2, 3, 4 to see some of the advantages and drawbacks of using alternative loss objectives for regression.

$\endgroup$
3
  • 2
    $\begingroup$ The classification rule you cite is only optimal under symmetric evaluation loss. If false positive and false negative have different costs to the end user, the probability cutoff of 0.5 is no longer optimal for classification. A further nuance that may be important here is the difference between estimation/training loss and prediction/evaluation loss, as has been discussed here. $\endgroup$ Commented Mar 14, 2023 at 20:15
  • $\begingroup$ True. When I say that $\eta$ is optimal, I mean with respect to the 0/1 loss (as in the Wiki article). Thanks for raising a good point. I think there could be something to say about regularization as well, but I wanted to keep this brief. $\endgroup$ Commented Mar 14, 2023 at 20:21
  • 1
    $\begingroup$ I suggest to make your definition of optimal explicit, as otherwise the statement only applies in a very special case and is generally incorrect. $\endgroup$ Commented Mar 14, 2023 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.