2
$\begingroup$

Let's say I'm using the Sonar data and I'd like to make a hold-out validation in R. I partitioned the data using the createFolds from caret package as folds <- createFolds(mydata$Class, k=5).

I would like then to use exactly the fold mydata[i] as test data and train a classifier using mydata[-i] as train data.

My first thought was to use the train function, but I couldn't find any support for hold-out validation. I mean, I don't want to partition my data and then test all the combinations for estimating the mean accuracy, I want to train it only once. Am I missing something here?

Also, I'd like to be able to use exactly the pre-defined folds as parameter, instead of letting the function partition the data. Does anyone have any thoughts?

Thanks in advance

$\endgroup$
3
$\begingroup$

It is hard to tell exactly what you are interested in. If it is a single hold-out, you can use

trainControl(method = "LGOCV", p = .8, number = 1)

and 80% will be used for training.

There is also method = "none" that will just fit the model for a single tuning parameter value (using the entire training set).

Also, if you want to use your own hold-out set(s), see the index argument of trainControl.

Max

$\endgroup$
1
  • $\begingroup$ It worked!! Thanks a lot, @topepo, and great job with the package! I noted, however, that if i pass an index with the method="knn", it gives an error. Is that expected? $\endgroup$
    – gcolucci
    Apr 10 '14 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.