I have 30,000 samples with 150 features for a binary classification problem, now I plan to follow: https://stackoverflow.com/questions/23815938/recursive-feature-elimination-and-grid-search-using-scikit-learn, wherein I'll do a GRIDSEARCHCV on RFECV, wherein the data is split twice in each fold (once for GRIDSEARCH and the other for RFE). I plan to do this on the ENTIRE dataset. Now, after determining the best features and parameters, I plan to build the final model by splitting the SAME dataset (the one Ive used for gridsearch and rfe) into the usual 70:15:15 training/validation/test set and use only the best features and parameters. My reason for this methodology is that the GRIDSEARCH and RFE was a different process from building the final model, and that there is no way for the final model to know what the GRIDSEARCH + RFE has learned on the data. Is this methodology valid or correct?

If not, I plan on splitting the dataset 50:50, wherein the first 50% would be for the gridsearch + rfe process, and the remaining 50% is for building the final model using the determined best features and parameters.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.