25
$\begingroup$

I have problem with defining the unit of accuracy in a regression task.

In classification tasks is easy to calculate sensitivity or specificity of classifier because output is always binary {correct classification, incorrect classification}. So I can count good/bad answers and based on the confusion matrix calculate some measurements.

But in regression tasks the output is a number. So I can't just say is it correct/incorrect -- instead I should measure "how far from true solution am I".

So what be the unit of accuracy in regression task?

Improve this question
$\endgroup$
8
  • 4
    $\begingroup$ Have you heard of coefficient of determination, a.k.a. $R^2$? That's a popular measure. $\endgroup$
    – Richard Hardy
    Mar 22, 2015 at 13:31
  • 3
    $\begingroup$ Have you looked here: en.wikipedia.org/wiki/Forecasting#Forecasting_accuracy ? $\endgroup$
    – 404pio
    Mar 22, 2015 at 15:16
  • 1
    $\begingroup$ If you're looking at the relative "goodness" of two or more models, you could also look into AIC or BIC $\endgroup$
    – Chris C
    Mar 22, 2015 at 17:02
  • 1
    $\begingroup$ @RichardHardy the $R^2$ can be useful but should definitely not be blindly suggested. I think the answer to the question depends on what is important in the regression. A often used approach in regression is to minimize the MSE. A downside of the $R^2$ is that when we add another factor into our regression the $R^2$ cannot go down even if the variable essentially adds no information. $\endgroup$
    – Jan
    Oct 18, 2018 at 12:42
  • 1
    $\begingroup$ @Jan, good points. I suggested to the OP looking at $R^2$ simply because it is the standard even though it is neither always relevant nor flawless. Good that you added some points of warning. It is also true that not all criticism of $R^2$ hits the point. For example, you may or may not worry about the fact that $R^2$ can only grow by adding more variables. It depends on what you are looking for. $\endgroup$
    – Richard Hardy
    Oct 18, 2018 at 13:14

1 Answer 1

Reset to default
29
$\begingroup$

You should ask yourself what were you trying to achieve with your modeling approach.

As you correctly said "how far from true solution am I" is a good starting point (notice this is also true for classification, we only get into specifics when we run into dichotomization, usually in more CS oriented machine learning, such as trees or SVMs).

So, let's measure it, shall we? If $x_i$ is the truth and $\hat x_i$ your model output, for sample $i$, here's the error:

$$\epsilon_i = x_i - \hat x_i$$

You could measure the mean error $\sum_i \epsilon_i$, but it turns out that, doing that, positive and negative errors cancel, giving you no way to know how good your model actually performs!

So, what people do in general, is to use these measures:

  • Squared error:

    $$\text{SE}=\sum_i^n \epsilon_i^2$$

  • Mean squared error:

    $$\text{MSE}=1/n \times \text{SE}$$

  • Root mean squared error:

    $$\text{RMSE}=\sqrt{\text{MSE}}$$

  • Relative mean squared error (do not confuse this for the RMSE, root mean squared error):

    $$\text{rMSE}={n-1\over n}{{\sum_i^n \epsilon_i^2}\over {\sum_i^n (x_i - \mathbb E(x))^2}}= {\text{MSE} \over Var(x)}$$

  • $\text{R}^2$:

    $$\text{R}^2=1 - \text{rMSE}$$

  • Absolute error:

    $$\text{AE}=\sum_i^n \sqrt{\epsilon_i^2}=\sum_i^n |{\epsilon_i}|$$

  • Mean absolute error:

    $$\text{MAE}=1/n \times \text{AE}$$

And many, many others. You can find them around on the site (see for example How to interpret error measures?).

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ (+1), but there is a typo in the MSE equation $\endgroup$
    – knrumsey
    Oct 17, 2018 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.