# Evaluating a regression model's performance using training and test sets?

I often hear about evaluating a classification model's performance by holding out the test set and training a model on the training set. Then creating 2 vectors, one for the predicted values and one for the true values. Obviously doing a comparison allows one to judge the performance of the model by its predictive power using things like F-Score, Kappa Statistic, Precision & Recall, ROC curves etc.

How does this compare to evaluating numeric prediction like regression? I would assume that you could train the regression model on the training set, use it to predict values, then compare these predicted values to the true values sitting in the test set. Obviously the measures of performance would have to be different since this isn't a classification task. The usual residuals and $R^2$ statistics are obvious measures but are there more/better ways to evaluate the performance for regression models? It seems like classification has so many options but regression is left to $R^2$ and residuals.

• I am not sure exactly what question you are asking but an obvious error metric for a regression model with a continuous output is mean square error (MSE) between the model output and the outcome variable. – BGreene Feb 24 '14 at 17:31
• So just an error measure between the actual and predicted. – StatTime Feb 24 '14 at 18:35
• Yes, optimized on the training set and validated using the test set. – BGreene Feb 25 '14 at 8:01

• Good answer. These are all associated with the second moment of the distribution. You can also look at the sum of differences if you are trying to eliminate bias, or use any combination you want. For example, $err = A\sum (x - x_i) + B\sum (x - x_i)^2$ where A and B are chosen weights for each scoring method. Really it will depend on what factors are important to your specific problem. – Greg Petersen Feb 25 '16 at 15:15