I have a dataset of 1931 observations and I intend to predict a binary outcome out of that. There is a list of 128 predictors (both binary and continuous). First I ran logistic regression modeling using all predictors and got a highly significant model (AUC = 0.84). Assuming that the high value of AUC was due to overfitting the model by using too many predictors, I did stepwise modeling to find the effective predictors:

mylogit <- glm(outcome ~ . , data = temp,family="binomial")
step <- step(mylogit, direction="both")

Now, I am not sure whether should I have done cross validation before or after stepwise modeling.

  • 1
    $\begingroup$ See also here & here. Any kind of outcome-based model selection has to be repeated as part of each training fold to get a fair estimate of the out-of sample performance of the whole procedure. $\endgroup$ Apr 22, 2015 at 8:53


Browse other questions tagged or ask your own question.