3
$\begingroup$

I have read the help manual of caret carefully: see A Short Introduction to the caret Package.

In its example, I found it split the data with createDataPartition before a model training.

library(caret)
library(mlbench)
data(Sonar)
set.seed(107)
inTrain <- creatDataPartition(y = Sonar$Class, p = .75, list = FALSE)
str(inTrain)
training <- Sonar[inTrain,]
testing <- Sonar[-inTrain,]

And does it make the repeated 10-fold CV with only the 75% data in the training set?

ctrl <- trainControl(method = "repeatedcv", repeats = 3, classProbs = TRUE, summaryFunction = twoClassSummary)
plsFit <- train(Class~., data = training, method = "pls", tuneLength = 15, trControl = ctrl, metric = "ROC", preProc =  c("center", "scale"))
plsFit

plsClass <- predict(plsFit, newdata = testing)

So, I think a k-fold CV make the "k-1" data into the training set and "1" data into the testing set, and all sets of data should be put into the testing data set at least once?

The code above uses only the 75% data in the whole model training process and the 10-fold CV process, and the rest (25%) only in the last prediction process. Do I understood these codes correctly?

I think the process in the caret manuals is different from the principal idea of k-fold CV, which really confused me.

And I want to perform 10-fold CV for model comparison, including rpart, adaboost, bagging, svm and kknn. Can the functions in caret make sense?
How can I do it?

$\endgroup$
0

1 Answer 1

4
$\begingroup$

The first bit:

inTrain <- creatDataPartition(y = Sonar$Class, p = .75, list = FALSE)

splits the totality of your data into training (75%) and test (25%).

Usually, cross-validation and other resampling methods are used on the training set. So if you use caret's train function for example and ask for 10-fold CV, the 10% held back is 10% of the training set.

So, I think a k-fold CV make the "k-1" data into the training set and "1" data into the testing set,

Sort of (the terminology is getting in the way). During each iteration of 10 fold CV, 90% is used to fit the model and 10% is held out for prediction. Again, these are percentages of your training set.

Can the functions in caret make sense?

You will have to clarify this part. train can fit many different models.

$\endgroup$
4
  • $\begingroup$ Thanks, you really help me a lot. And I still feel confuesd, why did it use the total data into the training set, so it can make k-fold CV for all of it and can make precidtion for the whole data. Or, only does the accuracy of prediction which is performed for hold out data will make sense? $\endgroup$
    – Archer
    Nov 17, 2013 at 2:20
  • $\begingroup$ @Archer, you split the data into train set (75%) and test set or called hold-out/back set (25%) as topepo said exactly. Finally, you should be able to optimize the model by using the train set only but that should be done via 10k-cv may be. so suppose you have 1000 whole data, 750 train, 250 test, tuning/optimization of model via e.g., 10-cv (very common approach) then you end up with 90% = 675 for fitting, the 10% = 75 for evaluation of the tuning parameters (specific to the model you choose). The best model finally evaluated on test = 250. Last thing fit model on data (10000). $\endgroup$
    – doctorate
    Jan 8, 2014 at 17:59
  • $\begingroup$ while tuning via 10k-cv you include repetitions like 5 to 10, so totally you would do tuning on 50 to 100 data folds each fold is 90% of the train set, and evaluation (on the 10%) would be more stable by now since it would be built upon 50 to 100 samples. Final best model, is the winner out of these times. $\endgroup$
    – doctorate
    Jan 8, 2014 at 18:03
  • $\begingroup$ @topepo: I still don't understand how \$results, \$bestTune, or \$finalModel contribute to $\hat y$ when I do predict.lm(). See stats.stackexchange.com/questions/142386/… . I assume \$results is my i*k matrix of $\hat\beta_1...\hat\beta_p$ and $\hat {(RMSE/R^2)}_{validate}$, but ?? $\endgroup$
    – jtd
    Mar 21, 2015 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.