0
$\begingroup$

I have time series dataset and as a requirement I prepare a 10 fold cross validation dataset. Briefly, I split data into 10 and for each training set I put a gap between last, first day of validation set and training set. That is, suppose that I take first 100 rows as validation. Then I take training set between the row range (150 - 5000). So 100-150 rows are removed. And repeated the same logic for other 9 folds. In the end I collected folds into dictionary like train[0], test[0] for fold_0, train[1], test[1] for fold_1 and so on...

My question is that can I introduce this specific cross val set into ensemble model in h2o. The example code in h2o's web site offers to use fold_column. But I could not figure it out how to use it in this case.

One another thing I have tried is that for each fold I have trained the data and get the predictions. And in the end I put 10 predictions into one list as below:

preds = []

preds.append([pred1,pred2,pred3,pred4,pred5,pred6,pred7,pred8,pred9,pred10])

So preds list has the same pattern as returned by

cross_validation_predictions()

But this time I get 10 different base models for each base model.

This is the link I am following and I need to ensemble regression base models.

http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/stacked-ensembles.html

So how should I proceed?

$\endgroup$
1
$\begingroup$

H2O Stacked Ensemble requires k-fold cross-validation of the base models and what you're describing (although it's close) isn't exactly k-fold cross-validation, so it does not support that. There is a ticket open for a modification of stacking that allows more flexibility, but it's still not exactly what you're describing.

To use the fold_column argument in H2O algorithms, you must have a column of fold indices. There is an example of how to construct that column in this Python unit test.

However, I am not sure that you really want to do k-fold cross-validation, or the variant you're describing, if you have time-series data. This article describes why that doesn't work (see the "Time series" paragraph). Usually with time-series data, you must ensure that you're always training on "past" data and predicting on "future" data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.