What is the theoretical justification for alternatives to MSE minimisation?

Question

I'm trying to wrap my head around the connection between statistical regression and its probability theoretical justification. In many books on statistics/machine learning, one is introduced to the idea of the loss function, which is then typically followed by a phrase of the flavour 'a popular choice for this function is mean squared loss'. As far as I understand, the justification for this choice stems from the theorem that

$$ \arg\min_{Z \in L^2(\mathcal{G})} \ \mathbb{E} \left[ (X - Z)^2 \right] = \mathbb{E} \left[ X \Vert \mathcal{G} \right] \tag{1} $$

where $X$ is the random variable to be estimated based on the information contained in $\mathcal{G}$. As far as I understand, probability theory teaches us that the conditional expectation $\mathbb{E}[X \Vert \mathcal{G}]$ is the best such estimate. If that's the case, why should our loss function still be a choice? Clearly we should be statistically estimating $\mathbb{E}[X \Vert \mathcal{G}]$, which by (*) implies minimizing the MSE.

An answer which I have often read is that we simply define the conditional expectation to satisfy (1), but that's doesn't seem true, as we have conditional expectations for any random variable in $L^1$. More importantly, there exists an intuitive theoretical explanation for why this definition gives us an estimator capturing all the information available after observing $\mathcal{G}$: we're using $\mathcal{G}$ to partition the total probability into possible paths and averaging over the remaining randomness in each of these. This interpretation in terms of information and $\sigma$-algebras has nothing to do, as far as I can tell, with minimizing MSE, we could have come up with it without ever knowing (1).

So my question really is: does minimizing MSE represent the theoretically optimal criterion, and if so, are we saying that any alternative (such as LAD) inherently represents a loss of theoretical optimality in favour of good estimation properties, etc.? Are necessarily leaving (as the explanation in the previous paragraph suggests) information contained in $\mathcal{G}$ on the table? And how do we quantify 'how much information' of $\mathcal{G}$ an estimator based on a different criterion (say, the median in case of LAD) utilises?

I've asked this question already on Mathematics Stack Exchange but I'm still not completely satisfied, so I was hoping someone here could maybe illuminate me. Judging by the number of similar questions on this subject, this is probably a phase all students of statistics pass through.

Answer 1

Depending on the situation, minimizing square loss might not even be the best way to estimate the mean. For instance, if data are Laplace-distributed, minimizing absolute loss gives an estimator that is, in many regards, better than minimizing square loss. Both are unbiased, but the minimization of absolute loss results in lower variance. This can be seen in a simulation.

library(ggplot2)
library(VGAM)
library(quantreg)
set.seed(2023)
N <- 10
R <- 5000
mse_based <- mae_based <- rep(NA, R)
for (i in 1:R){
  
  y <- VGAM::rlaplace(N, 0, 1)
  
  L <- lm(y ~ 1)
  Q <- quantreg::rq(y ~ 1, tau = 0.5)
  
  mse_based[i] <- summary(L)$coef[1, 1]
  mae_based[i] <- summary(Q)$coef[1]
  
  if (i %% 100 == 0){
    print(paste(
      i/R*100, 
      "% complete", 
      sep = ""
    ))
  }
  
}

d_mse <- data.frame(
  Estimate = mse_based,
  Estimation = "MSE Minimization"
)
d_mae <- data.frame(
  Estimate = mae_based,
  Estimation = "MAE Minimization"
)
d <- rbind(d_mse, d_mae)
ggplot(d, aes(x = Estimate, fill = Estimation)) +
  geom_density(alpha = 0.2) +
  theme(legend.position="bottom")

Visually, both have means around zero, which is the true value, but the MSE minimization results in considerably more variability in the estimates. This is not just a fluke of this simulation but can be shown mathematically for a Laplace distribution.

This is just one situation where an alternative to MSE minimization is theoretically justified.

Additionally, I demonstrate here that minimizing MSE need not produce a better out-of-sample MSE than minimizing MAE. This could be another justification for minimizing MAE instead of MSE.

Finally, in "classification" problems like logistic regression, making the (reasonable) assumption of a Bernoulli conditional distribution means that minimizing the log-loss, not the square loss, corresponds with maximum likelihood estimation, giving another theoretical justification for minimizing a loss function other than square loss.

Answer 2

When dealing with (binary) classification problems, e.g. problems where we are trying to map an input $x\in\mathcal X$ to a label $y\in\{+1;1\}$, it is easy to prove that the optimal classification rule (i.e. the classifier which minimizes the 0-1 loss) is given by $$\eta(x) = \text{sign}\ \mathbb E[Y\mid X = x] $$ Where the pair $(X,Y)$ follows the data distribution.

It thus makes sense to try to minimize the MSE $\mathbb E[(Y-f(X))^2] $, since as you said, the approximate minimizer $\hat f_{MSE}$ will be an approximation of $\mathbb E[Y\mid X] $ and therefore $\hat\eta_{MSE}:=\text{sign}\ \hat f_{MSE} $ should be a good approximation of the optimal classifier $\eta$ as well.

What we know, however, is that there exists a large family of surrogate losses $\ell$, such that any minimizer $f_\ell$ of $\mathbb E[\ell(f(X),Y)]$ satisfies $$\text{sign}f_{\ell} = \eta $$ Such loss functions are called Bayes consistent or classification calibrated, and there is an easy characterization for them and many of the practically used losses have this property (you can have a look here for more details). Concretely, this means that minimizing the loss associated to such an $\ell$ will solve our problem just as well as minimising the MSE. Hence, if the optimization problem associated with $\ell$ is easier to solve or has other desirable properties (regularization, robustness...) it makes sense to chose that option over minimizing the MSE.

---

The above discussion is mainly about (binary) classification, but even for that problem, some popular loss functions used in practice, such as cross-entropy, do not come with the theoretical guarantee mentioned above. Nevertheless, it happens to be very useful and exhibit many desirable properties (nicer loss landscape, stability during training...) and it arises naturally by considering the problem through another angle (look at the wikipedia page for more details).

Similarly, for regression, we may sometime prefer to minimize other metrics, such as the MAE or the MAPE rather than MSE, whose minimizers we know won't converge to the "optimal predictor", simply because we prefer our estimators to be more robust, depending on what we know about our data, or because we know that it will be more efficient computationally.

You can have a look at these threads : 1, 2, 3, 4 to see some of the advantages and drawbacks of using alternative loss objectives for regression.