Generally speaking, can RMSE be smaller than MAE?

I am calculating RMSE and MAE for my results. In two out of five methods, the RMSE is smaller than MAE. Note that I am using the same data, the same script, and the same code to calculate RMSE and MAE. The only difference is the algorithm to create the models.

I have been over my script several times and everything is ok with the code.

Thank you.

  • $\begingroup$ Generally speaking, yes. It depends on algorithms result. Different algorithms - different results. And check your data for outliers, maybe they effect on this measures. $\endgroup$ – ooolllooo Oct 12 '17 at 11:55
  • $\begingroup$ Can you expand a bit here by editing to say under what circumstances they might vary? $\endgroup$ – mdewey Oct 12 '17 at 12:21
  • $\begingroup$ For example, true values is 1,2,3 $\endgroup$ – ooolllooo Oct 12 '17 at 12:39
  • $\begingroup$ Does this answer stats.stackexchange.com/questions/59493/… help you move further with your problem? $\endgroup$ – mdewey Oct 12 '17 at 12:43
  • $\begingroup$ No, RMSE is always greater or equal MAE. Take a look at this article and this. $\endgroup$ – lenhhoxung Apr 20 '19 at 6:22

Both metrics are defined in terms of residuals

$$ r_i = y_i - \hat y_i $$

i.e. the difference between true values $y_i$ and predicted values $\hat y_i$. RMSE is defined as $\sqrt{1/n \sum_{i=1}^n r_i^2}$ and MAE as $1/n \sum_{i=1}^n |r_i|$. So answer to your question reduces to asking if squared values can be smaller then absolute values? To answer this yourself, plot both functions.

Squared vs absolute values

As you can see, for $x < 1$ it is true that $x^2 < |x|$, for $x = 1$, $x^2 = |x|$ and for $x > 1$, $x^2 > |x|$. So you can easily imagine such combination of values that taken together and averaged leads to smaller MAE then RMSE. Here you can find simple interpretation of such case.

| cite | improve this answer | |
  • $\begingroup$ This illustration is not correct because in RMSE, you take the average of the SSE and then calculate the root. See my answer below. $\endgroup$ – lenhhoxung Apr 20 '19 at 6:25

We should normally have $$MAE \leq RMSE$$

We can derive this (in a similar way as 'square of the mean values < mean of the squared values') with the triangle inequality or Jensen's inequality by expressing the absolute value of the error terms $\vert r_i \vert$ as a sum of two components: the mean of the absolute value of the error terms and the variation relative to that mean, $\vert r_i \vert = \mu + \delta_{i}$.

We can also express it explicitly without use of the triangle inequality

$$\begin{array}{} (r_1^2 + r_2^2 + \dots + r_n^2) &=& \vert r_1 \vert^2 + \vert r_2\vert^2 + \dots + \vert r_3 \vert^2 \\ & = & (\mu+\delta_1)^2 + (\mu+\delta_2)^2 +\dots + (\mu+\delta_n)^2\\ & = & n \mu^2 + 2 \mu (\delta_1 + \delta_2 +\dots + \delta_n) + \delta_1^2 + \delta_2^2 + \dots + \delta_n^2\\ & = & n \mu^2 + \delta_1^2 + \delta_2^2 + \dots + \delta_n^2 \leq n\mu^2 \end{array}$$

The step where $2 \mu (\delta_1 + \delta_2 +\dots + \delta_n)$ is removed comes from $\overline{\delta_{\vert r_i \vert}}=0$. Note that this must be the case for $\mu$ to be the mean of ${\vert r_i \vert}$, since $\overline{\vert r_i \vert} = \overline{\mu + \delta_i} = \overline{\mu} + \overline{\delta_i} = \mu + \overline{\delta_i}$)

Possible exception for different definition of 'mean'

Normally you compute the mean operation in RMSE and MAE by division with $n$ and then you get that the above inequality becomes

$$\frac{(r_1^2 + r_2^2 + \dots + r_n^2)}{n} \leq \mu^2 = \left( \frac{\vert r_1\vert + \vert r_2\vert + \dots + \vert r_n\vert }{n} \right)^2$$

But it is possible that you use a division by $n-1$ or $n-p$. And you compare

$$\frac{(r_1^2 + r_2^2 + \dots + r_n^2)}{n-p} \quad\text{versus} \quad \left( \frac{\vert r_1\vert + \vert r_2\vert + \dots + \vert r_n\vert }{n-p} \right)^2$$

In this case it is possible that the right side is smaller than the left side. Maybe this is the case for your code where you mention different methods to compute MAE and RMSE.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.