1
$\begingroup$

I am building a logistic regression model and am using k-fold cross-validation for model selection.

My doubt is with regards to misclassification rate. For measuring that, i will have to first find the optimal cut-off point from the ROC curve. For each individual model the cut-off value could be different.

So should i use the general rule-of thumb 0.5 has the cut-off during cross-validation to compare between the models.

Once the model with lowest error rate is identified, should i then find the optimal cut-off point for that model along?

$\endgroup$
2
$\begingroup$

The important observation here is that, given that you want to tune a cut off parameter to produce a specific misclassification rate, that parameter is part of your model. Said a slightly different way, choosing the cut off parameter is part of fitting your model. Cross validation should be aware of the entirety of your model fitting procedure.

In practice, you need to formalize a procedure for tuning this cutoff parameter. If you want to tune your cutoff parameter to produce a certain misclassification rate on your training data, then do so for every fold of your cross validation and evaluate the results on the out of sample fold. As you observe, you will get different cut off parameters for each fold, but this is no issue. You are evaluating a procedure at this point, not a final model. You can then make model selection type decisions confident that you are taking into account the full scope of your model fitting procedure.

When you have finished model selection decisions, you may follow the specified procedure on your entire training set, and then evaluate the final model's out of sample performance on a final hold out.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.