Use of previously set cross validation set for ensemble models in h2o

Question

I have time series dataset and as a requirement I prepare a 10 fold cross validation dataset. Briefly, I split data into 10 and for each training set I put a gap between last, first day of validation set and training set. That is, suppose that I take first 100 rows as validation. Then I take training set between the row range (150 - 5000). So 100-150 rows are removed. And repeated the same logic for other 9 folds. In the end I collected folds into dictionary like train[0], test[0] for fold_0, train[1], test[1] for fold_1 and so on...

My question is that can I introduce this specific cross val set into ensemble model in h2o. The example code in h2o's web site offers to use fold_column. But I could not figure it out how to use it in this case.

One another thing I have tried is that for each fold I have trained the data and get the predictions. And in the end I put 10 predictions into one list as below:

preds = []

preds.append([pred1,pred2,pred3,pred4,pred5,pred6,pred7,pred8,pred9,pred10])

So preds list has the same pattern as returned by

cross_validation_predictions()

But this time I get 10 different base models for each base model.

This is the link I am following and I need to ensemble regression base models.

http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/stacked-ensembles.html

So how should I proceed?

Answer 1

H2O Stacked Ensemble requires k-fold cross-validation of the base models and what you're describing (although it's close) isn't exactly k-fold cross-validation, so it does not support that. There is a ticket open for a modification of stacking that allows more flexibility, but it's still not exactly what you're describing.

To use the fold_column argument in H2O algorithms, you must have a column of fold indices. There is an example of how to construct that column in this Python unit test.

However, I am not sure that you really want to do k-fold cross-validation, or the variant you're describing, if you have time-series data. This article describes why that doesn't work (see the "Time series" paragraph). Usually with time-series data, you must ensure that you're always training on "past" data and predicting on "future" data.