1
$\begingroup$

It appears that I have completely misunderstood cross validation for several months so I want your help clarifying the idea using an example rather than only theory from some great SE questions. My fundamental misunderstanding comes from looking at the code. Coming off of this article, it's apparent that you're using cv to see how good the performance of the theoretical final model is. But using a cv method from train control, is that best model internally selected?

TL;DR: what exactly is a "final model"?

See my example for more:

attach(sat.act)
sat.act<- na.omit(sat.act)

#rename outcome and make as factor
sat.act <- sat.act %>% mutate(gender=ifelse(gender==1,"male","female"))
sat.act$gender <- as.factor(sat.act$gender)



#create train and test
indexes<-createDataPartition(y=sat.act$gender,p=0.7,list=FALSE)
train<-sat.act[indexes,]
test<-sat.act[-indexes,]


#set up RF
ctrl <- trainControl(method = "cv",
                  number = 5,
                  savePredictions = TRUE,
                  summaryFunction = twoClassSummary,
                  classProbs = TRUE)

model <- train(gender ~ ., data=train, 
              trControl = ctrl, 
              method= "rf", 
              preProc=c("center","scale"), 
              metric="ROC",
              importance=TRUE)

print(model) #some lines omitted below
ROC was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 3.

> model$resample
        ROC      Sens      Spec Resample
1 0.5861751 0.8225806 0.2285714    Fold1
2 0.6845351 0.8064516 0.3529412    Fold3
3 0.4717742 0.7580645 0.1764706    Fold2
4 0.4817362 0.7419355 0.2647059    Fold5
5 0.6930876 0.8709677 0.4000000    Fold4

If we look at model we see that it looked at three different mtrys and picked 3. And model$resample gives the results of the test folds. But what does that mean for the final model that is predicting on the test data?

# predict the outcome on a test set
model_pred <- predict(model, test)

# compare predicted outcome and true outcome
confusionMatrix(model_pred, test$gender)

So is model_pred splitting up the test in five parts and then predicting each of the five folds in model, just like how it did it when we trained the model? OR, is model_pred saying I already found the best mtry which was the only reason I did cv, so now when I predict, I'm using all of train and creating basically a one-fold cv problem to predict on test? Like does model_pred have it's own version of resample? I always though the latter but I don't think this is correct.

Basically if we manually pulled the indexes out and trained five models and then predicted on the remaining fold using predict , does that give you the same exact thing as that model$resample?

$\endgroup$
8
  • 1
    $\begingroup$ Have you taken a look at the results of model$finalModel? $\endgroup$ Jun 11, 2020 at 3:27
  • $\begingroup$ Yes I checked it out, but it's not really clear to me what is going into the final model. $\endgroup$
    – PleaseHelp
    Jun 11, 2020 at 14:47
  • 1
    $\begingroup$ OK, let me try to answer what is going on here and if it still isn't clear I'll write a real answer. The folds in CV are used to estimate out of sample error. For each combination of parameters, fit a model using 4/5 of the data and predict on the last 1/5. Record the error on that prediction. Do that for each fold. The final model is fit on all the data using the parameters which yielded the best score (e.g. highest ROC). Calling predict uses the best parameters to make a single prediction for each observation. The folds are not used at prediction time, only in training. $\endgroup$ Jun 11, 2020 at 15:15
  • $\begingroup$ thanks for the great explanation, I think I'm 80% of the way to understanding! So the final part I want to clarify: when predict is being used what does that actual model look like? I imagine something along the lines of ctrl <- trainControl(method = "none") and model <- train(gender ~ ., data=train, trControl = ctrl, method= "rf", tuneGrid = data.frame(mtry = 3)) (if we're pulling the mtry from above example)? I know you would not need to write this line in real life but for my understanding I'm trying to visualize all the steps that are happening in that 1 line of caret code. $\endgroup$
    – PleaseHelp
    Jun 11, 2020 at 15:33
  • 1
    $\begingroup$ More or less, yes. The train function is really a wrapper for the randomForest function. $\endgroup$ Jun 11, 2020 at 15:57

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.