I want to build a machine learning model using the caret package in R. Some of the features in my dataset are dummies taking the value 0 or 1. I would like to know which resampling methods can be used in the presence of dummy variables.

k-fold cross-validation does not seem to be an option as explained in this post. I could potentially use leave-one-out cross-validation; however, this seems to be too expensive when N is large (my dataset has 100000+ observations).

Are there any suitable resampling methods in the presence of dummies and large N?

  • 2
    $\begingroup$ (+1) "Not an option" is rather strong - if there aren't too few 1s (or 0s) it'll be very unlikely to get none in an 80% or 90% sample. See also stratified cross-validation. $\endgroup$ – Scortchi - Reinstate Monica Sep 6 '16 at 18:08
  • 2
    $\begingroup$ Thanks for the useful hint regarding stratiefied CV! I actually do have some very rare groups in my sample (lots of 0s) and, having looked at stratified CV, this seems to be a promising approach to deal with this issue, i.e. making sure that the training data includes also data on the rare class. A good discussion on the topic can be found here $\endgroup$ – kanimbla Sep 6 '16 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.