2
$\begingroup$

I am working on a binary classification using random forest with a dataset size of 977 records and 6 columns. class ratio is 77:23 (imbalanced dataset)

Since, my dataset is small, I learnt that it is not advisable to split using regular train_test split of 70 and 30.

So, I was thinking to do repeatedKfold CV. Please find my code below

Approach 1 - Full data - X, y

rf_boruta = RandomForestClassifier(class_weight='balanced',max_depth=3,max_features='sqrt',n_estimators=300)
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=100)
scores = cross_val_score(rf_boruta,X,y, scoring='f1', cv=cv)
print('mean f1: %.3f' % mean(scores))

But I see that we have full input data X passed at once to the model. Doesn't this lead to data leakage? Meaning, if I am doing categorical encoding, we have to do based on all categories encountered in full dataset. Similarly, consider if a dataset ranges from the year 2017 to 2022. It is possible that model uses 2021 data in one of the folds and validate it on the 2020 data.

So, is it right to use repeatedKfold like the below?

Approach 2 - only train data - X_train, y_train

rf_boruta = RandomForestClassifier(class_weight='balanced',max_depth=3,max_features='sqrt',n_estimators=300)
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=100)
scores = cross_val_score(rf_boruta,X_train,y_train, scoring='f1', cv=cv)
print('mean f1: %.3f' % mean(scores))

I ask because the tutorials that I see online here and here all use full dataset (X,y) as input to RepeatedKfold

Can help me understand which will be the best approach to use?

$\endgroup$

1 Answer 1

2
$\begingroup$

The KFold is used to estimate the performance, not to tune the hyper-parameters. Therefore, you should use the full dataset, and it does not cause data leakage. The test is always performed on a holdout set. Repeating this procedure multiple times is just Repeated KFold. Note that KFold CV is different than time series CV, and it assumes no temporal dependency between the samples.

$\endgroup$
3
  • $\begingroup$ thanks for your response. Upvoted. Yes, I am usimg Kfold to estimate performance. So, for my concern of temporal dependency, I think timeseries CV is better $\endgroup$
    – The Great
    Apr 3, 2022 at 23:16
  • $\begingroup$ or do you think as shown in my approach 2, wht if I split my X into X_train and X_test based on time? Meanimg, X_train will have all records <= 2020 and X_test will have all records > 2020? Do you think that will work kfold? Where I only pass X_train to kfoldcv? $\endgroup$
    – The Great
    Apr 3, 2022 at 23:17
  • $\begingroup$ Yes, that works for testing. Why do yo do cross validation? I don’t see HP tuning here. If it is for performance evaluation, you do not need to do kfold if you have a separate test set. $\endgroup$
    – gunes
    Apr 6, 2022 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.