# How do you validate your machine learning models?

I am wondering what approaches are commonly used for validating machine learning models designed for classification or prediction tasks:

Approaches that am using at the moment:

Using truth-sets: - ROCs, Bootstrapping, Accuracy, Sensitivity, Specificity, Cross-validation

Orthogonal validation: - Use a different class of algorithm that can perform prediction or classification task and compare results

Any other suggestions ?

If you have an independent holdout sample this is all easy to do. Otherwise I recommend the optimism bootstrap to bias- (overfitting-) correct the apparent calibration curve to account for regression to the mean. This is implemented for parametric models in the R rms package's calibrate function.
Sensitivity, specificity, and ROC curves do not play a role here, although in the simple binary $Y$ case the area under the ROC curve is a simple linear translation of a particular rank correlation measure: Somers' $D_{xy} = 2\times (c - \frac{1}{2})$ where the $c$-index is the ROC area in the simple case.
• The R rms package validate and calibrate functions handle the usual regression models. If you have an independent sample validation, the val.prob function does this. – Frank Harrell Sep 7 '14 at 13:08
• So if you get predicted probabilities from RF you can use val.prob for independent sample validation or as part of a cross-validation loop. – Frank Harrell Sep 7 '14 at 13:46