1
$\begingroup$

I am trying to better understand how changing a threshold affects a cross validation model. So if you trained a random forest model, the default threshold is threshold=0.5. And I understand that if the predicted result gets a score >0.5 it is considered a positive case and vice versa. But if you have a 5-fold cross validation model, is the model looking at what happens in the first four folds, and then looks at the threshold in order to give you the results on the test fold OR does it apply the threshold on just the test fold? Or in other words, what does the threshold change? The results of the training folds or just of the testing fold?

I guess more technically speaking, looking at the example below, it appears that the results given for each fold is for the testing fold. So does that mean the threshold is evaluated on the testing fold and the training folds make don't care about the threshold?

attach(iris)
#create a binary outcome on Sepal.Length
iris <- iris %>% mutate(Sepal.Length=ifelse(Sepal.Length>5.0,"aff","neg")) 


ctrl <- trainControl(method="cv", 
                     number=5,
                     summaryFunction=twoClassSummary, 
                     classProbs=T,
                     savePredictions = T)

model <- train(Sepal.Length ~ ., data = iris, trControl = ctrl, method= 
                    "rf", preProc=c("center","scale"), metric="ROC",importance=TRUE, tuneGrid = 
                    data.frame(mtry = 2)) 

#examine outcome at every fold
print(model$pred)

> print(model$pred)
    pred obs   aff   neg rowIndex mtry Resample
#1    aff neg 0.616 0.384        7    2    Fold1
#2    neg neg 0.116 0.884       10    2    Fold1
#3    aff aff 0.602 0.398       15    2    Fold1
#4    aff aff 0.894 0.106       19    2    Fold1
#5    aff neg 0.706 0.294       25    2    Fold1
#6    aff neg 0.716 0.284       27    2    Fold1
#7    neg neg 0.020 0.980       43    2    Fold1
#8    neg neg 0.034 0.966       48    2    Fold1
#9    aff aff 1.000 0.000       51    2    Fold1
#10   aff aff 1.000 0.000       60    2    Fold1
$\endgroup$
1
$\begingroup$

But if you have a 5-fold cross validation model, is the model looking at what happens in the first four folds, and then looks at the threshold in order to give you the results on the test fold OR does it apply the threshold on just the test fold?

You train a model based on the first 4 folds, then this model predicts the probabilities for the test fold. With this probability, the test fold are assigned labels and this is checked against the actual labels, with a 0.5 cutoff, to calculate the accuracy, ROC etc.

Or in other words, what does the threshold change? The results of the training folds or just of the testing fold?

The threshold doesn't change. At least not in the example you showed. If you want to change the threshold to make a prediction, its another thing altogether and has nothing to do with cross-validation. This involves re-writing a predict() function that sets the cutoff at something and incorporating this into randomForest() and train() .

It appears that the results given for each fold is for the testing fold.

To add to the above, yes every data point in your data has a chance to be in the test fold and be predicted by the other 4 folds.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ So to recapitulate, it seems like the threshold affects the final prediction of the test fold (either aff or neg in this case). So if you are modifying the threshold to get a certain value for sensitivity for your overall model which will predict on separate data, then does this mean you would want each of your training folds to have that sensitivity or you would want that trained model to have that sensitivity (i.e. average of all testing folds)? $\endgroup$ – PleaseHelp Jun 1 at 22:03
  • $\begingroup$ Ok I think I get what you mean. So you are thinking of using another threshold for sensitivity calculation and your question is whether you can train your model on that? $\endgroup$ – StupidWolf Jun 1 at 22:06
  • $\begingroup$ Yeah I want to make a model with sensitivity=0.5 and I don't know what's more appropriate: train a model and then modify model$pred so it uses the threshold I predetermined which would then give me sen=0.5 for the final model. And that means all folds are evaluated on the same threshold but it also means each "set" (train fold1-4, test fold 5, etc) has a different sensitivity ind sen that averages to sen=0.5. Or the alternative, train each set so each has sen=0.5 but the overall average of test folds is unlike to be sen=0.5. I think the latter undoes cv since each fold is different $\endgroup$ – PleaseHelp Jun 1 at 22:16
  • $\begingroup$ sensitivity = TP/(TP+FN), how do you force a model to stop at 0.5 ? I think this is a bit contorted now.. So first of all, when you train a model, you usually choose that which has the highest defined score, ROC, accuracy etc. Now you have a very weird case, so you can choose the one, that is closest to 0.5 sensitivity $\endgroup$ – StupidWolf Jun 1 at 22:26
  • $\begingroup$ I was basically following along [this] (stats.stackexchange.com/questions/221409/…) to find the threshold which uses the ROC curve to give me an overall sensitivity of 50%. I guess I'm trying to conceptually understand how it does that, and you make it sound like it does it for each of the individual test folds to get that overall sen=0.5. But now I'm not sure if it is right to affect the threshold on the training folds or the testing folds if that makes sense? $\endgroup$ – PleaseHelp Jun 1 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.