I am stacking various models (Gradient Boosting Machines, Random Forests, Linear Regressions) using a k-fold cross validation for the train set $X_{train}$, therefore obtaining out-of-sample prediction over the whole train set ($X_{train}'$).

I am using the whole training set before predicting on the test (held out) set.

Then, I am combining my models using a simple penalized regressions over $X_{train}'$ and I observe the performance on the test set.

The question is : should the folds used for the train set be the same for every model ?

I observed a gain in performance when using different folds (both in cross-validation in the training set and on the held out set) but I have no idea why this is happening.


1 Answer 1


Personally, I would not do that. I would not run just one sweep either.

I would run the process 100 different times, using different random starts, and look at the distribution over the folds. If I have a pathological problem it shows up by having that "one magic sample" that has perfect performance or that "one evil sample" that has horrible performance, and I don't wait to trip over them by luck. I run it dozens of times and get a sense of the spread.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.