How do we decide whether mean absolute error or mean square error is better for linear regression? Are there other loss functions that are commonly used for linear regression?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Caution: when you speak of "linear regression" most people will assume you are minimizing squared error loss. The general estimator $\hat{\beta} = \text{argmin}_\beta \text{ } q(Y-X\beta)$ where $q$ is any convex loss is called minimax. $\endgroup$– AdamODec 3, 2018 at 15:36
-
$\begingroup$ I have on rare occasion fit data to the lowest sum of squared relative error. This was done for calibration of industrial metal thickness gauges. $\endgroup$– James PhillipsDec 3, 2018 at 16:41
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
Put simply: it matters what error metric matters most to you. I have not personally seen a useful application of absolute error loss. In 99% of cases, people use squared error loss. Regression, by definition, is about modeling trend lines that approximate a mean response over a range of predictors.
If the CLT applies, that mean response tends to a normal distribution. Squared error loss minimax estimator of the mean gives the MLE for normally distributed data. This means that asymptotically, you will get correct 95% CI and p-values for the statistical tests about model parameters.
Even the small sample performance of squared error loss is surprising and favorable, based on my experience. Robust error estimation by bootstrap is usually not profoundly different, or when it is, it seems to be a result of small sample sizes in which both methods perform poorly ("No Free Lunch" theorem for statistics).
In a somewhat pathological alternate example, if the errors are double exponential, the absolute error minimax estimator of the mean gives the MLE. There are some strange sorts of counterexamples to the CLT where the asymptotic distribution of the test statistic tends toward exponential (Huzurbazar), and thus with a mixture could be double exponential.
A concluding remark is one of efficiency. In the case of mean estimation: the central limit theorem says the rate of convergence to the limiting distribution is root N. This is because $\sqrt{n} \left( \hat{\theta}_{L_2} - \theta \right) \rightarrow_d $ a non-singular distribution. As @whuber points out in the comments below, the absolute error loss yields the median as the optimal measure of central tendency in a univariate estimation case. Like the mean, the median also tends to a normal distribution as $n \rightarrow \infty$ but at a much lower rate. In fact, $n^{1/4} \left( \hat{\theta}_{L_1} - \theta \right) \rightarrow d $ a non-singular distribution. So we say that the median is a root-root-n consistent estimator whereas the mean is a root-n consistent estimator: the mean goes to what it's estimating much faster, and thus provides more powerful tests.
Hence, if a solution is to be offered on strictly statistical terms, I would prefer squared error loss not just for it's predominant usage, but for it's theoretically sound probability model for the residuals given correct mean model specification.
-
4$\begingroup$ Re "have not ... seen a useful application:" Absolute error loss corresponds to quantile regression at the median. This seems to be a frequently used robust procedure. It's arguable that the CLT almost never applies to regression problems and asymptotics are irrelevant because by the time the sample size becomes that large, you shouldn't be doing OLS regression anyway! $\endgroup$– whuber ♦Dec 3, 2018 at 16:15
-
$\begingroup$ @whuber the OP did not ask about quantile regression, rather it was linear regression. This concerns the means not the quantiles of the conditional distribution of the response. Also, in what sense does the CLT never apply? Further, how and why would asymptotics be irrelevant? Maybe we can clarify by stating the objective of the analysis. I stated this implicitly in my answer, but to make explicit: I consider the objective to be modeling the conditional mean response of the outcome as a function of one or more predictors. $\endgroup$– AdamODec 3, 2018 at 16:27
-
$\begingroup$ @AdamO is there any good book or sources that cover the whole series of theories behind these? $\endgroup$– user181372Dec 4, 2018 at 1:25
-
$\begingroup$ Adam, "linear" regression methods include quantile regression. And it scarcely depends on what definition of "linear" you mean. The CLT is unlikely to apply because to apply it you would need a large number of replications for each combination of regressors. When that happens, you have so much information that you can posit much richer models, obviating the need for the simpler linear models in the first place. You have in effect defined away a whole set of constructive answers by insisting that regression aims at estimating a conditional mean rather than some other conditional parameter. $\endgroup$– whuber ♦Dec 4, 2018 at 2:08
-
$\begingroup$ @sweetyBaby One good book? Likely no, but I've been reading Bayesian and Frequentist Regression Methods by Jon Wakefield quite a bit and the section on non-parametric methods may be of interest. $\endgroup$– AdamODec 4, 2018 at 15:38