Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
For classification problems I have been using Neural Networks and measuring Type I and II error using the confusion matrix and its measures as per this resource (mirror), which is pretty straight forward.
When faced with an estimation problem, how would one assess the model performance? Assuming that there are no classes and the output is interpreted in real form. Beyond averaging distance metrics, which does not lend much insight.
The link that you posted has many of the techniques that I would suggest, but additionally plotting learning curves can help. This can help you see not just the absolute performance, but can help you get a sense of how far from optimal performance you are.
Learning Curves: If you plot cross-validation (cv) error and training set error rates versus training set size, you can learn a lot. If the two curves approach each other with low error rate, then you are doing well.
If it looks like the curves are starting to approach each other and both heading/staying low, then you need more data.
If the cv curve remains high, but the training set curve remains low, then you have a high-variance situation. You can either get more data, or use regularization to improve generalization.
If the cv stays high and the training set curve comes up to meet it, then you have high bias. In this case, you want to add detail to your model.
There is multiple ways to define performance criteria of model in estimation. Most of people use how good the model fit the data. So in case of regression it will be "how much of variance is explain by the model". However, you need to be careful with such regression when you are performing variable selection (for eg. by LASSO) you need to control for the number of parameter are included in the model. One can use cross-validated version of explained variance which presumably give unbiased estimate model performance.
First of all, I think you should use the term "regression" or "prediction" instead of "estimation" - the latter rather refers to statistical inference for model parameters (assuming some parametric form), whereas you seem to be more concerned with predictive power for dependent variable. Now, from my consulting experience, most often used measures of model performance - apart from the simplest "distance metrics" you mention - are realtive mean absolute/squared error and $R^2$ coefficient for observed and predicted values. Of course you can use some custom loss functions, depending on a particular study/business context.
Referring to scikit-learn documentation (Python based package for machine learning), r2_score and explained_variance_score are popular choices. Unlike distance measures like mean_squared_error or mean_absolute_error, these metrics give an indication of how good or bad the prediction is (closer to 1 => better predictions). [By the way, if using distance measures, I would recommend RMSE (root mean square error) instead of just MSE (mean square error) so that the magnitude can be compared with the predictions]
Alternatively, you could also compute correlation coefficient between regressor predicted values and the true target variable values using Pearson's correlation coefficient (for linear models) or better go for Spearman's rank correlation coefficient (as this doesn't assume linear models and is less sensitive to outliers).
Learning curves suggested in John Yetter's reply is also a good method but the above mentioned metrics might be easier to assess the performance.
