113
$\begingroup$

Why use Root Mean Squared Error (RMSE) instead of Mean Absolute Error (MAE)??

I've been investigating the error generated in a calculation - I initially calculated the error as a Root Mean Normalised Squared Error.

Looking a little closer, I see the effects of squaring the error gives more weight to larger errors than smaller ones, skewing the error estimate towards the odd outlier. This is quite obvious in retrospect.

In what instance would the Root Mean Squared Error be a more appropriate measure of error than the Mean Absolute Error? The latter seems more appropriate to me or am I missing something?

To illustrate this I have attached an example below:

  • The scatter plot shows two variables with a good correlation,

  • the two histograms to the right chart the error between Y(observed ) and Y(predicted) using normalised RMSE (top) and MAE (bottom).

plots comparing RMSE and MAE

There are no significant outliers in this data and MAE gives a lower error than RMSE. Is there any rational, other than MAE being preferable, for using one measure of error over the other?

Improve this question
$\endgroup$
6
  • 17
    $\begingroup$ Because RMSE and MAE are two different measures of error, a numerical comparison between them (which is involved in asserting that MAE is "lower" than RMSE) does not seem meaningful. That line must have been fit according to some criterion: that criterion, whatever it is, must be the relevant measure of error. $\endgroup$
    – whuber ♦
    Jan 22, 2013 at 18:33
  • $\begingroup$ the line was fitted using least squares - but the pic is just an example to show the difference in measured error. My real issue is in using an optimiser to solve for four function parameters to some measure of minimised error, MAE or RMSE. $\endgroup$
    – user1665220
    Jan 22, 2013 at 18:47
  • $\begingroup$ Thank you for the clarification. But what error are you interested in, precisely? The error in the fit or the errors in the parameter estimates? $\endgroup$
    – whuber ♦
    Jan 22, 2013 at 18:48
  • 1
    $\begingroup$ The error in the fit. I have some lab samples that give y, which I want to predict using a function. I optimise the function for 4 exponents by minimising the error for the fit between the observed and predicted data. $\endgroup$
    – user1665220
    Jan 22, 2013 at 18:57
  • $\begingroup$ In RMSE we consider the root of number of items (n). That is root of MSE divided by root of n. Root of MSE is ok, but rather than dividing by n it is divided by root of n to receive RMSE. I am feeling that it would be a policy. Reality would be (Root of MSE)/n. In that way MAE is better. $\endgroup$
    – user21700
    Mar 8, 2013 at 0:11

5 Answers 5

Reset to default
101
$\begingroup$

This depends on your loss function. In many circumstances it makes sense to give more weight to points further away from the mean--that is, being off by 10 is more than twice as bad as being off by 5. In such cases RMSE is a more appropriate measure of error.

If being off by ten is just twice as bad as being off by 5, then MAE is more appropriate.

In any case, it doesn't make sense to compare RMSE and MAE to each other as you do in your second-to-last sentence ("MAE gives a lower error than RMSE"). MAE will never be higher than RMSE because of the way they are calculated. They only make sense in comparison to the same measure of error: you can compare RMSE for Method 1 to RMSE for Method 2, or MAE for Method 1 to MAE for Method 2, but you can't say MAE is better than RMSE for Method 1 because it's smaller.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I understand that MAE will never be higher than RMSE. I have been using both error estimates and looking at the difference between values to give an indication as to the impact of outliers. I.e when they are close great, when they further apart i investigate to see whats going on. Ultimately i want to predict parameters that best suit the data, and e.g. 9% error sound better than 12% - i just wanted to make sure i'm picking the right one for the right reason. Cheers for your advice $\endgroup$
    – user1665220
    Jan 22, 2013 at 17:45
  • 1
    $\begingroup$ The main difference between RMSE (consequently MSE) and MAE is not about how they weight errors. You can use a weight function if needed. The main difference is that MSE is related to L2 Space (MAE has no such thing). So for example, MSE could measure amount of energy needed for a closed loop control when E is the feedback signal (Remember Mean Square of a signal ,Error in this case, is proportional to it's energy). Also so much of mathematics and consequently algorithms like Marquardt-Levenberg works in this space. simply put, they use MSE as their objective function. $\endgroup$
    – eulerleibniz
    Dec 6, 2018 at 23:11
36
$\begingroup$

Here is another situation when you want to use (R)MSE instead of MAE: when your observations' conditional distribution is asymmetric and you want an unbiased fit. The (R)MSE is minimized by the conditional mean, the MAE by the conditional median. So if you minimize the MAE, the fit will be closer to the median and biased.

Of course, all this really depends on your loss function.

The same problem occurs if you are using the MAE or (R)MSE to evaluate predictions or forecasts. For instance, low volume sales data typically have an asymmetric distribution. If you optimize the MAE, you may be surprised to find that the MAE-optimal forecast is a flat zero forecast.

Here is a little presentation covering this, and here is a recent invited commentary on the M4 forecasting competition where I explained this effect.

Improve this answer
$\endgroup$
6
  • $\begingroup$ +1. The idea of comparing distributions is great but... wouldn't a metric like the one you present fail miserably in something like N = 1e3; set.seed(1); y = rpois(N, lambda=1); yhat = c(y[2:N],0)? The predictive densities "difference" would be minimal but the actual yhat would be useless. Granted, this is an extreme case. (I might be missing something obvious, apologies for that in advance - I do not have access to the paper just the presentation.) $\endgroup$
    – usεr11852
    Aug 13, 2017 at 21:29
  • $\begingroup$ @usεr11852: yes, your sequence of point forecasts would be useless, and in particular, much worse than a flat forecast $\hat{y}=1$ (which is both the mean and the median, so it's optimal for both MAE and MSE). A density forecast is not just a sequence of point forecasts! It's a complete density prediction for each future time point. So we would predict a Pois(1) for the first time point, for the second, for the third etc. $\endgroup$
    – Stephan Kolassa
    Aug 14, 2017 at 6:30
  • $\begingroup$ Thank you very much for the clarifications; I can conceptualise the presentation better now. (Hmm... I need to get hold of your paper after all. :) ) $\endgroup$
    – usεr11852
    Aug 14, 2017 at 17:51
  • $\begingroup$ @usεr11852: feel free to contact me by email (find the address here) - if your mail doesn't end up in my spam filter, I'll happily send you that paper. $\endgroup$
    – Stephan Kolassa
    Aug 15, 2017 at 20:06
  • $\begingroup$ @usεr11852 I completely lost you after "like N=" what that is? $\endgroup$
    – sak
    Jul 16, 2019 at 18:12
14
$\begingroup$

enter image description here

RMSE is a more natural way of describing loss in Euclidean distance. Therefore if you graph it out in 3D, the loss is in a cone shape, as you can see above in green. This also applies to higher dimensions, although it's harder to visualize it.

MAE can be thought of as city-block distance. It isn't really as natural of a way to measure loss, as you can see in the graph in blue.

Improve this answer
$\endgroup$
4
$\begingroup$

To put it in short, if there are many outliers then you may consider using Mean Absolute Error (also called the Average Absolute Deviation). RMSE is more sensitive to outliers than the MAE. But when outliers are exponentially rare (like in a bell-shaped curve), the RMSE performs very well and is generally preferred.

Both the RMSE and the MAE are ways to measure the distance between two vectors: the vector of predictions and the vector of target values. MAE corresponds to the l1 norm or Manhattan norm while RMSE corresponds to the l2 norm or Euclidian Norm. The higher the norm index, the more it focuses on large values and neglects small ones

Improve this answer
$\endgroup$
2
$\begingroup$

When prediction is less focal than parameter estimation, the Gauss-Markov theorem might be relevant:

In a linear model with spherical errors, OLS - the solution to the MSE minimization problem - is efficient in a class of linear unbiased estimators - there are (restrictive, to be sure) conditions under which "you can't do better than OLS".

I am not arguing this should justify using OLS almost all of the time, but it sure contributes to why (especially since it is a good excuse to focus so much on OLS in teaching).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.