# How to interpret error measures?

I am running the classify in Weka for a certain dataset and I've noticed that if I'm trying to predict a nominal value the output specifically shows the correctly and incorrectly predicted values. However, now I'm running it for a numerical attribute and the output is:

Correlation coefficient                 0.3305
Mean absolute error                     11.6268
Root mean squared error                 46.8547
Relative absolute error                 89.2645 %
Root relative squared error             94.3886 %
Total Number of Instances               36441


How do I interpret this? I've tried googling each notion but I don't understand much since statistics is not at all in my field of expertise. I would greatly appreciate an ELI5 type of answer in terms of statistics.

Let's denote the true value of interest as $\theta$ and the value estimated using some algorithm as $\hat{\theta}$.

Correlation tells you how much $\theta$ and $\hat{\theta}$ are related. It gives values between $-1$ and $1$, where $0$ is no relation, $1$ is very strong, linear relation and $-1$ is an inverse linear relation (i.e. bigger values of $\theta$ indicate smaller values of $\hat{\theta}$, or vice versa). Below you'll find an illustrated example of correlation. Mean absolute error is:

$$\mathrm{MAE} = \frac{1}{N} \sum^N_{i=1} | \hat{\theta}_i - \theta_i |$$

Root mean square error is:

$$\mathrm{RMSE} = \sqrt{ \frac{1}{N} \sum^N_{i=1} \left( \hat{\theta}_i - \theta_i \right)^2 }$$

Relative absolute error:

$$\mathrm{ RAE} = \frac{ \sum^N_{i=1} | \hat{\theta}_i - \theta_i | } { \sum^N_{i=1} | \overline{\theta} - \theta_i | }$$

where $\overline{\theta}$ is a mean value of $\theta$.

Root relative squared error:

$$\mathrm{ RRSE }= \sqrt{ \frac{ \sum^N_{i=1} \left( \hat{\theta}_i - \theta_i \right)^2 } { \sum^N_{i=1} \left( \overline{\theta} - \theta_i \right)^2 }}$$

As you see, all the statistics compare true values to their estimates, but do it in a slightly different way. They all tell you "how far away" are your estimated values from the true value of $\theta$. Sometimes square roots are used and sometimes absolute values - this is because when using square roots the extreme values have more influence on the result (see Why square the difference instead of taking the absolute value in standard deviation? or on Mathoverflow).

In $\mathrm{ MAE}$ and $\mathrm{ RMSE}$ you simply look at the "average difference" between those two values - so you interpret them comparing to the scale of your valiable, (i.e. $\mathrm{ MSE}$ of 1 point is a difference of 1 point of $\theta$ between $\hat{\theta}$ and $\theta$).

In $\mathrm{ RAE}$ and $\mathrm{ RRSE}$ you divide those differences by the variation of $\theta$ so they have a scale from 0 to 1 and if you multiply this value by 100 you get similarity in 0-100 scale (i.e. percentage). The values of $\sum(\overline{\theta} - \theta_i)^2$ or $\sum|\overline{\theta} - \theta_i|$ tell you how much $\theta$ differs from it's mean value - so you could tell that it is about how much $\theta$ differs from itself (compare to variance). Because of that the measures are named "relative" - they give you result related to the scale of $\theta$.

Check also those slides.

• Thank you for your explanation! I am trying to evaluate the performance of various algorithms. So for example, if I get this other output (Correlation: 0.3044, MAE: 10.832, MSE: 47.2971, RAE: 83.163%, RSE: 95.2797%) and I try to compare it to the first one, which one could I say performed better? Jan 5 '15 at 15:18
• You should choose the model with bigger correlation and smaller error estimates. As you see, there are multiple measures of model performance (and those are only few them) and sometimes they give different answers. It is almost never the "yes/no" kind of answer you get. The task of model selection would get easier if you catch up with theory, you can check for example those lectures.
– Tim
Jan 5 '15 at 17:32
• Thank you very much! I went ahead and marked your reply as the answer because you've helped me plenty! Jan 6 '15 at 9:57
• @Tim Mean absolute error should probably be abbreviated as MAE:) Feb 16 '17 at 14:04
• @MewX What kind of references you are looking for? It is basically a rescaled RMSE. There is not much to say about it...
– Tim
Apr 29 '17 at 8:05