10
$\begingroup$

I often hear about evaluating a classification model's performance by holding out the test set and training a model on the training set. Then creating 2 vectors, one for the predicted values and one for the true values. Obviously doing a comparison allows one to judge the performance of the model by its predictive power using things like F-Score, Kappa Statistic, Precision & Recall, ROC curves etc.

How does this compare to evaluating numeric prediction like regression? I would assume that you could train the regression model on the training set, use it to predict values, then compare these predicted values to the true values sitting in the test set. Obviously the measures of performance would have to be different since this isn't a classification task. The usual residuals and $R^2$ statistics are obvious measures but are there more/better ways to evaluate the performance for regression models? It seems like classification has so many options but regression is left to $R^2$ and residuals.

$\endgroup$
  • 1
    $\begingroup$ I am not sure exactly what question you are asking but an obvious error metric for a regression model with a continuous output is mean square error (MSE) between the model output and the outcome variable. $\endgroup$ – BGreene Feb 24 '14 at 17:31
  • $\begingroup$ So just an error measure between the actual and predicted. $\endgroup$ – StatTime Feb 24 '14 at 18:35
  • $\begingroup$ Yes, optimized on the training set and validated using the test set. $\endgroup$ – BGreene Feb 25 '14 at 8:01
2
$\begingroup$

As said, typically, the Mean Squared Error is used. You calculate your regression model based on your training set, and evaluate its performance using a separate test set (a set on inputs x and known predicted outputs y) by calculating the MSE between the outputs of the test set (y) and the outputs given by the model (f(x)) for the same given inputs (x).

Alternatively you can use following metrics: Root Mean Squared Error, Relative Squared Error, Mean Absolute Error, Relative Absolute Error... (ask google for definitions)

$\endgroup$
  • $\begingroup$ Good answer. These are all associated with the second moment of the distribution. You can also look at the sum of differences if you are trying to eliminate bias, or use any combination you want. For example, $err = A\sum (x - x_i) + B\sum (x - x_i)^2$ where A and B are chosen weights for each scoring method. Really it will depend on what factors are important to your specific problem. $\endgroup$ – Greg Petersen Feb 25 '16 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.