I have a task with daily entries for which I need to do binary classification. Suppose I have 18 months of data and the model is refit every month. In addition I've got about 150 one-hot encoded features and 50 continuous (many features are lagged values).

I'd like to do:

  • Chronological, expanding 'walk forward' train/test split to create several folds that would most closely resemble the 'real-life' situation (model refit monthly):

    custom_iterable_cv = [ (X_train_1 = X[month 0: month 12], X_test_1 = X[month 13]), ... (X_train_6 = X[month 0: month 17], X_test_6 = X[month 18]) ]

  • Preprocessing on features (PCA on dummies, StandardScaling on continuous) applied as discussed in the literature and in numerous threads on this forum, including PCA and the train/test split and PCA and k-fold cross-validation in caret package in R and How to use test set data if model has been built using a training set transformed with PCA? and many others.




  • Do hyperparameter optimization and potentially feature selection using something like the sklearn tools GridSearchCV, RFECV, SelectFromModel or hyperopt.

Question: Are there any standard packages that implement this idea? sklearn cross-validators, grid search and feature selection tools (SelectFromModel, RFECV) do not seem to work. If negative, is there a smart way to implement this idea using existing building blocks from sklearn or other packages?

  • Without standard scaling and pca it is totally trivial using a custom cv=[(fold1 tuple),(fold2 tuple)...], but PCA and scaling seems necessary to reduce dimensionality and to fit linear models.

  • Without chronological expanding 'walk-forward' split and custom iterable it can be done as in PCA and cross-validation (even though it seems that in that case GridSearchCV selects best parameters based on 3-folds from the train data where each the PCA for each fold is not independent from other folds and this GridSearchCV overfits, but whatever), but there is time dependency and lagged features, chronological split seems necessary.

  • An sklearn pipeline and/or column transformer could be used to combine preprocessing and the classification estimator into another estimator that could be passed into a GridSearchCV along with the custom cv iterable, but that would refit the preprocessing to the test data which is only 1 month long.

Thus, none of these solutions works.

A complementary question for this case: Do I actually need to do nested feature selection since even after the PCA the number of features remains close to 100? Without PCA it is totally a must, since some of the dummies and continuous features would most probably turn out to be noise, but since PCA is also a way of denoising, I wonder whether it is a good idea.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.