0
$\begingroup$

I am working with the GBM algorithm in H2O in R. I am using 100% of the data as the training data, and then using 5-fold cross-validation to train and validate my model using 100% of the data.

My question, then, is how are the training metrics and the cross-validation metrics calculated? I'm fairly new to all this, so my understanding may be limited and I'm trying to wrap my head around these things.

I took a look at the description here: http://docs.h2o.ai/h2o/latest-stable/h2o-docs/cross-validation.html, and it seems to describe how the cross-validation metrics are calculated but not the training metrics:

"For the main model, this is how the cross-validation metrics are computed: The 5 holdout predictions are combined into one prediction for the full training dataset (i.e., predictions for every row of the training data, but the model making the prediction for a particular row has not seen that row during training). This “holdout prediction” is then scored against the true labels, and the overall cross-validation metrics are computed."

So is it the case that, to calculate the cross-validation metrics, the model will combine the predictions on each (i.e., n=5) held-out group of observations, and then compare those to the actual observed values? In that case, how are the training metrics computed? Any explanation would be helpful!

$\endgroup$
0
$\begingroup$

The training metrics are calculated as you would expect: using the predictions of the model on the training dataset compared to the actual predictions. Remember the final model is built on the entire dataset, there is no hold-out/validation dataset.

The description you pasted is meant to explain how it is that you could build a model on the entire dataset and still have cross-validation metrics. In the case of H2O-3, it does this by taking the hold-out predictions of each fold and then comparing those to the actual predictions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.