1
$\begingroup$

I am wondering what approaches are commonly used for validating machine learning models designed for classification or prediction tasks:

Approaches that am using at the moment:

Using truth-sets: - ROCs, Bootstrapping, Accuracy, Sensitivity, Specificity, Cross-validation

Orthogonal validation: - Use a different class of algorithm that can perform prediction or classification task and compare results

Any other suggestions ?

$\endgroup$
4
$\begingroup$

You did not specify the single most important thing to validate along with predictive discrimination (e.g., rank correlation between predicted and observed): calibration accuracy. The calibration curve can be estimated nonparametrically using loess. It is important that this demonstration of absolute predictive accuracy be done with high resolution, i.e., without binning predictions in any way.

If you have an independent holdout sample this is all easy to do. Otherwise I recommend the optimism bootstrap to bias- (overfitting-) correct the apparent calibration curve to account for regression to the mean. This is implemented for parametric models in the R rms package's calibrate function.

Sensitivity, specificity, and ROC curves do not play a role here, although in the simple binary $Y$ case the area under the ROC curve is a simple linear translation of a particular rank correlation measure: Somers' $D_{xy} = 2\times (c - \frac{1}{2})$ where the $c$-index is the ROC area in the simple case.

$\endgroup$
  • $\begingroup$ Thanks Frank for a great answer. Can you recommend an R package to do calibration accuracy ? Thanks again! $\endgroup$ – Khader Shameer Sep 7 '14 at 12:46
  • 1
    $\begingroup$ The R rms package validate and calibrate functions handle the usual regression models. If you have an independent sample validation, the val.prob function does this. $\endgroup$ – Frank Harrell Sep 7 '14 at 13:08
  • $\begingroup$ My current model is based on RF. Thanks again, Frank - big help! $\endgroup$ – Khader Shameer Sep 7 '14 at 13:22
  • 1
    $\begingroup$ So if you get predicted probabilities from RF you can use val.prob for independent sample validation or as part of a cross-validation loop. $\endgroup$ – Frank Harrell Sep 7 '14 at 13:46
  • $\begingroup$ How are you planning on obtained calibrated results from a RF? lots of people present risk models from RF, but still unclear to me if this makes sense. These people believe you can (but they don't know how to create a high resolution calibration curve?): ncbi.nlm.nih.gov/pubmed/21915433 $\endgroup$ – charles Sep 7 '14 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.