Model stacking, should the folds in the training set be the same?

Question

I am stacking various models (Gradient Boosting Machines, Random Forests, Linear Regressions) using a k-fold cross validation for the train set $X_{train}$, therefore obtaining out-of-sample prediction over the whole train set ($X_{train}'$).

I am using the whole training set before predicting on the test (held out) set.

Then, I am combining my models using a simple penalized regressions over $X_{train}'$ and I observe the performance on the test set.

The question is : should the folds used for the train set be the same for every model ?

I observed a gain in performance when using different folds (both in cross-validation in the training set and on the held out set) but I have no idea why this is happening.

Answer 1

Personally, I would not do that. I would not run just one sweep either.

I would run the process 100 different times, using different random starts, and look at the distribution over the folds. If I have a pathological problem it shows up by having that "one magic sample" that has perfect performance or that "one evil sample" that has horrible performance, and I don't wait to trip over them by luck. I run it dozens of times and get a sense of the spread.