53
$\begingroup$

Why use Root Mean Squared Error (RMSE) instead of Mean Absolute Error (MAE)??

Hi

I've been investigating the error generated in a calculation - I initially calculated the error as a Root Mean Normalised Squared Error.

Looking a little closer, I see the effects of squaring the error gives more weight to larger errors than smaller ones, skewing the error estimate towards the odd outlier. This is quite obvious in retrospect.

So my question - in what instance would the Root Mean Squared Error be a more appropriate measure of error than the Mean Absolute Error? The latter seems more appropriate to me or am I missing something?

To illustrate this I have attached an example below:

  • The scatter plot shows two variables with a good correlation,

  • the two histograms to the right chart the error between Y(observed ) and Y(predicted) using normalised RMSE (top) and MAE (bottom).

enter image description here

There are no significant outliers in this data and MAE gives a lower error than RMSE. Is there any rational, other than MAE being preferable, for using one measure of error over the other?

$\endgroup$

migrated from stackoverflow.com Jan 22 '13 at 17:13

This question came from our site for professional and enthusiast programmers.

  • 9
    $\begingroup$ Because RMSE and MAE are two different measures of error, a numerical comparison between them (which is involved in asserting that MAE is "lower" than RMSE) does not seem meaningful. That line must have been fit according to some criterion: that criterion, whatever it is, must be the relevant measure of error. $\endgroup$ – whuber Jan 22 '13 at 18:33
  • $\begingroup$ the line was fitted using least squares - but the pic is just an example to show the difference in measured error. My real issue is in using an optimiser to solve for four function parameters to some measure of minimised error, MAE or RMSE. $\endgroup$ – user1665220 Jan 22 '13 at 18:47
  • $\begingroup$ Thank you for the clarification. But what error are you interested in, precisely? The error in the fit or the errors in the parameter estimates? $\endgroup$ – whuber Jan 22 '13 at 18:48
  • 1
    $\begingroup$ The error in the fit. I have some lab samples that give y, which I want to predict using a function. I optimise the function for 4 exponents by minimising the error for the fit between the observed and predicted data. $\endgroup$ – user1665220 Jan 22 '13 at 18:57
  • $\begingroup$ In RMSE we consider the root of number of items (n). That is root of MSE divided by root of n. Root of MSE is ok, but rather than dividing by n it is divided by root of n to receive RMSE. I am feeling that it would be a policy. Reality would be (Root of MSE)/n. In that way MAE is better. $\endgroup$ – user21700 Mar 8 '13 at 0:11
54
$\begingroup$

This depends on your loss function. In many circumstances it makes sense to give more weight to points further away from the mean--that is, being off by 10 is more than twice as bad as being off by 5. In such cases RMSE is a more appropriate measure of error.

If being off by ten is just twice as bad as being off by 5, then MAE is more appropriate.

In any case, it doesn't make sense to compare RMSE and MAE to each other as you do in your second-to-last sentence ("MAE gives a lower error than RMSE"). MAE will never be higher than RMSE because of the way they are calculated. They only make sense in comparison to the same measure of error: you can compare RMSE for Method 1 to RMSE for Method 2, or MAE for Method 1 to MAE for Method 2, but you can't say MAE is better than RMSE for Method 1 because it's smaller.

$\endgroup$
  • $\begingroup$ I understand that MAE will never be higher than RMSE. I have been using both error estimates and looking at the difference between values to give an indication as to the impact of outliers. I.e when they are close great, when they further apart i investigate to see whats going on. Ultimately i want to predict parameters that best suit the data, and e.g. 9% error sound better than 12% - i just wanted to make sure i'm picking the right one for the right reason. Cheers for your advice $\endgroup$ – user1665220 Jan 22 '13 at 17:45
  • $\begingroup$ The main difference between RMSE (consequently MSE) and MAE is not about how they weight errors. You can use a weight function if needed. The main difference is that MSE is related to L2 Space (MAE has no such thing). So for example, MSE could measure amount of energy needed for a closed loop control when E is the feedback signal (Remember Mean Square of a signal ,Error in this case, is proportional to it's energy). Also so much of mathematics and consequently algorithms like Marquardt-Levenberg works in this space. simply put, they use MSE as their objective function. $\endgroup$ – eulerleibniz Dec 6 '18 at 23:11
15
$\begingroup$

Here is another situation when you want to use (R)MSE instead of MAE: when your observations' conditional distribution is asymmetric and you want an unbiased fit. The (R)MSE is minimized by the conditional mean, the MAE by the conditional median. So if you minimize the MAE, the fit will be closer to the median and biased.

Of course, all this really depends on your loss function.

The same problem occurs if you are using the MAE or (R)MSE to evaluate predictions or forecasts. For instance, low volume sales data typically have an asymmetric distribution. If you optimize the MAE, you may be surprised to find that the MAE-optimal forecast is a flat zero forecast.

Here is a little presentation covering this, and here is a recent paper I wrote on the sales forecasting aspect.

$\endgroup$
  • $\begingroup$ +1. The idea of comparing distributions is great but... wouldn't a metric like the one you present fail miserably in something like N = 1e3; set.seed(1); y = rpois(N, lambda=1); yhat = c(y[2:N],0)? The predictive densities "difference" would be minimal but the actual yhat would be useless. Granted, this is an extreme case. (I might be missing something obvious, apologies for that in advance - I do not have access to the paper just the presentation.) $\endgroup$ – usεr11852 Aug 13 '17 at 21:29
  • $\begingroup$ @usεr11852: yes, your sequence of point forecasts would be useless, and in particular, much worse than a flat forecast $\hat{y}=1$ (which is both the mean and the median, so it's optimal for both MAE and MSE). A density forecast is not just a sequence of point forecasts! It's a complete density prediction for each future time point. So we would predict a Pois(1) for the first time point, for the second, for the third etc. $\endgroup$ – Stephan Kolassa Aug 14 '17 at 6:30
  • $\begingroup$ Thank you very much for the clarifications; I can conceptualise the presentation better now. (Hmm... I need to get hold of your paper after all. :) ) $\endgroup$ – usεr11852 Aug 14 '17 at 17:51
  • $\begingroup$ @usεr11852: feel free to contact me by email (find the address here) - if your mail doesn't end up in my spam filter, I'll happily send you that paper. $\endgroup$ – Stephan Kolassa Aug 15 '17 at 20:06
  • $\begingroup$ @usεr11852 I completely lost you after "like N=" what that is? $\endgroup$ – sak yesterday
5
$\begingroup$

enter image description here

RMSE is a more natural way of describing loss in Euclidean distance. Therefore if you graph it out in 3D, the loss is in a cone shape, as you can see above in green. This also applies to higher dimensions, although it's harder to visualize it.

MAE can be thought of as city-block distance. It isn't really as natural of a way to measure loss, as you can see in the graph in blue.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.