H2o k-fold validation [closed]

Question

I need to get some clarification on how H2o creates a training model from the k-fold validations. Below is my understanding, please correct where I am wrong:

• If I set nfolds = 5, then H2o will split the training data into 5 hold out data sets (20% each) that are distinct (no repetition).
• The remaining 80% of the training data is used for building the model and evaluated on the hold out (20%). This is repeated 5 times.
• Finally, the predictions from the 5 hold out data sets are pooled together to get back a 100% of the training data set and then a MAIN model is trained on this.
• This MAIN model is the one we use for prediction?

Answer 1

The main model is built on 100% of the training data, which is why when you set nfolds=5 you actually have 6 models built.

The documentation discusses the process here. The main section to review is:

"In general, for all algos that support the nfolds parameter, H2O’s cross-validation works as follows:

For example, for nfolds=5, 6 models are built.

The first 5 models (cross-validation models) are built on 80% of the training data, and a different 20% is held out for each of the 5 models.

Then the main model is built on 100% of the training data. This main model is the model you get back from H2O in R, Python and Flow (though the CV models are also stored and available to access later).

This main model contains training metrics and cross-validation metrics (and optionally, validation metrics if a validation frame was provided). The main model also contains pointers to the 5 cross-validation models for further inspection.

All 5 cross-validation models contain training metrics (from the 80% training data) and validation metrics (from their 20% holdout/validation data). To compute their individual validation metrics, each of the 5 cross-validation models had to make predictions on their 20% of of rows of the original training frame, and score against the true labels of the 20% holdout.

For the main model, this is how the cross-validation metrics are computed: The 5 holdout predictions are combined into one prediction for the full training dataset (i.e., predictions for every row of the training data, but the model making the prediction for a particular row has not seen that row during training). This “holdout prediction” is then scored against the true labels, and the overall cross-validation metrics are computed.

This approach has some implications. Scoring the holdout predictions freshly can result in different metrics than taking the average of the 5 validation metrics of the cross-validation models. For example, if the sizes of the holdout folds differ a lot (e.g., when a user-given fold_column is used), then the average should probably be replaced with a weighted average. Also, if the cross-validation models map to slightly different probability spaces, which can happen for small DL models that converge to different local minima, then the confused rank ordering of the combined predictions would lead to a significantly different AUC than the average."