I am running the classify in Weka for a certain dataset and I've noticed that if I'm trying to predict a nominal value the output specifically shows the correctly and incorrectly predicted values. However, now I'm running it for a numerical attribute and the output is:

Correlation coefficient                 0.3305
Mean absolute error                     11.6268
Root mean squared error                 46.8547
Relative absolute error                 89.2645 %
Root relative squared error             94.3886 %
Total Number of Instances               36441 

How do I interpret this? I've tried googling each notion but I don't understand much since statistics is not at all in my field of expertise. I would greatly appreciate an ELI5 type of answer in terms of statistics.

up vote 39 down vote accepted

Let's denote the true value of interest as $\theta$ and the value estimated using some algorithm as $\hat{\theta}$.

Correlation tells you how much $\theta$ and $\hat{\theta}$ are related. It gives values between $-1$ and $1$, where $0$ is no relation, $1$ is very strong, linear relation and $-1$ is an inverse linear relation (i.e. bigger values of $\theta$ indicate smaller values of $\hat{\theta}$, or vice versa). Below you'll find an illustrated example of correlation.

Correlation example

(source: http://www.mathsisfun.com/data/correlation.html)

Mean absolute error is:

$$\mathrm{MAE} = \frac{1}{N} \sum^N_{i=1} | \hat{\theta}_i - \theta_i | $$

Root mean square error is:

$$ \mathrm{RMSE} = \sqrt{ \frac{1}{N} \sum^N_{i=1} \left( \hat{\theta}_i - \theta_i \right)^2 } $$

Relative absolute error:

$$ \mathrm{ RAE} = \frac{ \sum^N_{i=1} | \hat{\theta}_i - \theta_i | } { \sum^N_{i=1} | \overline{\theta} - \theta_i | } $$

where $\overline{\theta}$ is a mean value of $\theta$.

Root relative squared error:

$$ \mathrm{ RRSE }= \sqrt{ \frac{ \sum^N_{i=1} \left( \hat{\theta}_i - \theta_i \right)^2 } { \sum^N_{i=1} \left( \overline{\theta} - \theta_i \right)^2 }} $$

As you see, all the statistics compare true values to their estimates, but do it in a slightly different way. They all tell you "how far away" are your estimated values from the true value of $\theta$. Sometimes square roots are used and sometimes absolute values - this is because when using square roots the extreme values have more influence on the result (see Why square the difference instead of taking the absolute value in standard deviation? or on Mathoverflow).

In $ \mathrm{ MAE}$ and $ \mathrm{ RMSE}$ you simply look at the "average difference" between those two values - so you interpret them comparing to the scale of your valiable, (i.e. $ \mathrm{ MSE}$ of 1 point is a difference of 1 point of $\theta$ between $\hat{\theta}$ and $\theta$).

In $ \mathrm{ RAE}$ and $ \mathrm{ RRSE}$ you divide those differences by the variation of $\theta$ so they have a scale from 0 to 1 and if you multiply this value by 100 you get similarity in 0-100 scale (i.e. percentage). The values of $\sum(\overline{\theta} - \theta_i)^2$ or $\sum|\overline{\theta} - \theta_i|$ tell you how much $\theta$ differs from it's mean value - so you could tell that it is about how much $\theta$ differs from itself (compare to variance). Because of that the measures are named "relative" - they give you result related to the scale of $\theta$.

Check also those slides.

  • Thank you for your explanation! I am trying to evaluate the performance of various algorithms. So for example, if I get this other output (Correlation: 0.3044, MAE: 10.832, MSE: 47.2971, RAE: 83.163%, RSE: 95.2797%) and I try to compare it to the first one, which one could I say performed better? – FloIancu Jan 5 '15 at 15:18
  • 5
    You should choose the model with bigger correlation and smaller error estimates. As you see, there are multiple measures of model performance (and those are only few them) and sometimes they give different answers. It is almost never the "yes/no" kind of answer you get. The task of model selection would get easier if you catch up with theory, you can check for example those lectures. – Tim Jan 5 '15 at 17:32
  • Thank you very much! I went ahead and marked your reply as the answer because you've helped me plenty! – FloIancu Jan 6 '15 at 9:57
  • 1
    @Tim Mean absolute error should probably be abbreviated as MAE:) – Antoine Feb 16 '17 at 14:04
  • 1
    @MewX What kind of references you are looking for? It is basically a rescaled RMSE. There is not much to say about it... – Tim Apr 29 '17 at 8:05

protected by Community Dec 3 '17 at 14:25

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.