How are the training and cross-validation metrics calculated in H2O?

Question

I am working with the GBM algorithm in H2O in R. I am using 100% of the data as the training data, and then using 5-fold cross-validation to train and validate my model using 100% of the data.

My question, then, is how are the training metrics and the cross-validation metrics calculated? I'm fairly new to all this, so my understanding may be limited and I'm trying to wrap my head around these things.

I took a look at the description here: http://docs.h2o.ai/h2o/latest-stable/h2o-docs/cross-validation.html, and it seems to describe how the cross-validation metrics are calculated but not the training metrics:

"For the main model, this is how the cross-validation metrics are computed: The 5 holdout predictions are combined into one prediction for the full training dataset (i.e., predictions for every row of the training data, but the model making the prediction for a particular row has not seen that row during training). This “holdout prediction” is then scored against the true labels, and the overall cross-validation metrics are computed."

So is it the case that, to calculate the cross-validation metrics, the model will combine the predictions on each (i.e., n=5) held-out group of observations, and then compare those to the actual observed values? In that case, how are the training metrics computed? Any explanation would be helpful!

Answer 1

The training metrics are calculated as you would expect: using the predictions of the model on the training dataset compared to the actual predictions. Remember the final model is built on the entire dataset, there is no hold-out/validation dataset.

The description you pasted is meant to explain how it is that you could build a model on the entire dataset and still have cross-validation metrics. In the case of H2O-3, it does this by taking the hold-out predictions of each fold and then comparing those to the actual predictions.