6
$\begingroup$

I am using Caret and have divided my data into training(75%) and test (25%) sets. Now I am running 10-Fold CV on training data. I fit the following model:

train_control <- trainControl(method="cv", number=10)
output <- train(Species~., data=iris, trControl=train_control, method="rpart2")

Question 1: Is the accuracy shown by output an average Sub-Test set or Sub-Training i.e. (10% of Training data or 90% of Training data)

Question 2: How can I view the accuracy of both sets for each folds i.e. accuracy of Sub-Test and Sub-Training set for each of 10-Folds

$\endgroup$

1 Answer 1

16
$\begingroup$

So using 10-fold CV will split the data into 10 different sets of roughly the same size. The model is fit on 90% and the remaining 10% is used to estimate accuracy. This process continues "round robin" 9 more times.

The accuracy is the average of the 10 holdouts for each tuning value. For example:

> set.seed(1)
> train_control <- trainControl(method="cv", number=10,
+                               savePredictions = TRUE) 
> output <- train(Species~., data=iris, trControl=train_control, method="rpart2")
note: only 2 possible values of the max tree depth from the initial fit.
 Truncating the grid to 2 .

There is a sub-object that has the hold-out estimates of accuracy for each fold, and averaging these gets you the reported value (the ones shown here are for the optimal value of Cp):

> output$resample
        Accuracy Kappa Resample
    1  0.8666667   0.8   Fold01
    2  0.9333333   0.9   Fold02
    3  1.0000000   1.0   Fold03
    4  0.9333333   0.9   Fold04
    5  0.9333333   0.9   Fold05
    6  0.8000000   0.7   Fold06
    7  1.0000000   1.0   Fold07
    8  1.0000000   1.0   Fold08
    9  0.9333333   0.9   Fold09
    10 0.9333333   0.9   Fold10
    > mean(output$resample$Accuracy)
[1] 0.9333333
> getTrainPerf(output)
  TrainAccuracy TrainKappa method
1     0.9333333        0.9 rpart2

There is little value in getting predictions on the 90% each time. Since the same data is used to build the model, the predictions can be extremely optimistic (which is the motivation for using cross-validation in the first place).

If you want to see where the estimates in output$resample were created:

> ## For the model associated with optimal Cp value, here is the predictions on the 
> ## first fold that was held-out
> first_holdout <- subset(output$pred, Resample == "Fold01")
> head(first_holdout)
        pred        obs rowIndex maxdepth Resample
1     setosa     setosa        6        2   Fold01
2     setosa     setosa       27        2   Fold01
3     setosa     setosa       31        2   Fold01
4     setosa     setosa       36        2   Fold01
5     setosa     setosa       45        2   Fold01
6 versicolor versicolor       67        2   Fold01

If we get the accuracy for this set:

> postResample(first_holdout$pred, first_holdout$obs)
 Accuracy     Kappa 
0.7666667 0.6500000 

Max

$\endgroup$
3
  • $\begingroup$ Thanks a lot!!!!. You have really helped me in getting to this point. Can you please share a link which gives the list of all sub-objects of the output? $\endgroup$
    – Malu
    Sep 4, 2014 at 5:46
  • $\begingroup$ It may not be 100% up to date, but look at the Value section of ?train $\endgroup$
    – topepo
    Sep 4, 2014 at 21:37
  • $\begingroup$ Thanks I got the list at this location inside-r.org/packages/cran/caret/docs/extractPrediction $\endgroup$
    – Malu
    Sep 5, 2014 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.