How do you validate your machine learning models?

Question

I am wondering what approaches are commonly used for validating machine learning models designed for classification or prediction tasks:

Approaches that am using at the moment:

Using truth-sets: - ROCs, Bootstrapping, Accuracy, Sensitivity, Specificity, Cross-validation

Orthogonal validation: - Use a different class of algorithm that can perform prediction or classification task and compare results

Any other suggestions ?

Answer 1

You did not specify the single most important thing to validate along with predictive discrimination (e.g., rank correlation between predicted and observed): calibration accuracy. The calibration curve can be estimated nonparametrically using loess. It is important that this demonstration of absolute predictive accuracy be done with high resolution, i.e., without binning predictions in any way.

If you have an independent holdout sample this is all easy to do. Otherwise I recommend the optimism bootstrap to bias- (overfitting-) correct the apparent calibration curve to account for regression to the mean. This is implemented for parametric models in the R rms package's calibrate function.

Sensitivity, specificity, and ROC curves do not play a role here, although in the simple binary $Y$ case the area under the ROC curve is a simple linear translation of a particular rank correlation measure: Somers' $D_{xy} = 2\times (c - \frac{1}{2})$ where the $c$-index is the ROC area in the simple case.