5
$\begingroup$

Several articles says that MAE is robust to outliers but MSE is not and MSE can hamper the model if errors are too huge. My question is that MSE and MAE both are error matrices, our priority is to just minimise the error whether we use MSE or MAE. Where does outliers come into play while using error matrices?

What difference can an error matrix make in linear regression for choosing optimal values of the parameters (in regards to outliers because as per my knowledge error matrices doesn't contribute for choosing parameters, its the loss function we are minimising) which we are trying to learn ($y=mx+c$ : parameters being $m$ and $c$)?

But if they are not helping with parameters then why are we worried about outliers while choosing error matrixes?

$\endgroup$
1
  • 1
    $\begingroup$ It can be off-topic, but you should remember that MSE vs MAE is about your data distribution. Since linear regression is y~Ax+b+\eps, where \eps is an error term and if you assume that \eps is Normally distributed then MLE turns into MSE minimization. If you replace Normal distribution with Laplace distribution then it will be minimization of MAE. $\endgroup$
    – l4morak
    Commented Jul 17, 2022 at 6:42

2 Answers 2

4
$\begingroup$

Since the error is defined $\hat{u}=y-\hat{y}$ (difference between actual and predicted value), taking the squared error will give a high weight to outliers (with "large" difference between $y-\hat{y}$). This will also affect the mean error.

Using the absolute difference between $y-\hat{y}$ will give outliers a lower weight compared to the case of taking squares.

Suppose $y=10$, $\hat{y}=110$, and $\hat{u}=-100$. We want to get rid of the negative sign, so we take squares or use the absolute value:

$\hat{u}^2=10,000$

$|\hat{u}|=100$

Suppose $y=10$, $\hat{y}=11$, and $\hat{u}=-1$. We want to get rid of the negative sign, so we take squares or use the absolute value:

$\hat{u}^2=1$

$|\hat{u}|=1$

MAE and MSE are simply different ways of weighting the errors. There are also alternatives, such as the Huber loss, which combines MSE and MAE by using MSE logic for "small" residual values and MAE logic for "large" residual values (which might have a huge impact).

enter image description here

Image taken from Little Bobby Tables's answer to this post on Stack Overflow.

$\endgroup$
1
  • $\begingroup$ With respect to the outliers, what if the difference is smaller than 1? Suppose, $y=0.5$ and $\hat{y}=5\cdot 10^{-10}$. Then because the difference $\hat{y} - y < 1$ then squaring the error will be smaller than the absolute value of error. $\endgroup$
    – ado sar
    Commented Sep 21, 2023 at 9:42
3
$\begingroup$

It depends on your problem, but I personally like MAE the most. Things to take into account:

  • Historically, MSE has been used instead of MAE because the math is easier to write and naturally appears in some constructions (e.g. euclidean distance). Also, when differentiating manually, the sign appears everywhere if you use MAE. This argument has lost power with computers.

  • Mathematically, MAE and MSE will give different results. For instance, the mean of the targets minimizes MSE, while the median minimizes the MAE. When the data is normally distributed, the mean is the best estimator of the center of the distribution you can get (it is the best linear unbiased estimator, the maximum likelihood estimator and minimum variance unbiased estimator). MAE works similarly for Laplace distributions.

  • Practically, MAE is more robust to outliers (as explained in another answer) and, if you have no reason to give extra importance to larger errors (which are already larger and more important), then MAE seems to me more agnostic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.