0
$\begingroup$

Is there a way to compare model performance in the caret package by combining the resampled predictions from every fold/repeat? I am working with a small (<1000 rows) and severely imbalanced (4% positives) data set and I am primarily concerned with precision (using prSummary as the summaryFunction). When I use 10-fold 5-repeat cross validation, the individual precision-recall curves seem to be unstable and a poor representation of model performance. Is it possible to use all of the resampled predictions to build a single curve (rather then the average of 50 curves) and compare it to models with different tuning parameters?

This is my first question on Cross Validated/Stack Exchange so please let me know if there are any other details I can provide.

Thanks!

$\endgroup$
  • $\begingroup$ I would suggesting trying something more like 100+ repeats of a bootstrap as usually CV procedures suffer from variance in small samples (as you experience yourself here). Conceding that a procedure is unstable with "only five(x10)" samples is a bit defeatist. Give it a bit more oomph! :) (Also consider stratifying your sample when doing CV or Bootstrap.) $\endgroup$ – usεr11852 Jan 30 '18 at 23:33
  • $\begingroup$ Thanks for the response. I think my question may have been a bit misleading. Increasing the number of repeats would just generate more precision-recall curves with 100 samples in the validation set. I want to evaluate the AUC for a PR curve that combines all of the validation sets (5000 samples, given 10-fold 5-repeats) rather than taking the average of 50 curves built on 100 samples $\endgroup$ – BU_ Jan 31 '18 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.