2
$\begingroup$

When a metric is computed for a trained model with nfolds greater than 1 in h2o, what is the value when `xval=TRUE' ?

h2o.mse(model, xval=TRUE)

Here is my read and please correct me if I'm off. If nfolds is 3, it is doing 3-fold cross validation to select the best model. So, the mse(train=T) gives the MSE of the best fold of the three models evaluated on the training set. Likewise, valid=TRUE gives the error on the validation set by the same model.

But what is the MSE when xval is selected?

According to the h2o.mse help page, xval is the cross-validation error. So I expect it is the average of the 3 cross-validation runs.

However, when i look at the actual values, xval is always lower than valid, which contradicted my interpretation above. The best model must have a lower MSE than the average.

$\endgroup$

migrated from stackoverflow.com Dec 22 '16 at 22:13

This question came from our site for professional and enthusiast programmers.

1
$\begingroup$

When you set nfolds>1, H2O will perform k-fold cross-validation for the specific purpose of model evaluation, not model selection. It will train k models for cross-validation and then a final model on the full training set. The value of h2o.mse(model, xval=TRUE) is the average MSE over the k models trained in the cross-validation process.

The value of h2o.mse(model, train=TRUE) will provide the (full) training set MSE and if you passed a separate validation_frame, then h2o.mse(model, valid=TRUE) will provide the validation MSE.

If you want actually perform model selection (by choosing the best model based on cross-validation error), then take a look at h2o.grid().

$\endgroup$
  • $\begingroup$ Yet another clarification Q. When valid or train=TRUE, which model of the nfolds models is used? or again it is the average of all the CV models. $\endgroup$ – horaceT Jan 3 '17 at 22:51
  • 1
    $\begingroup$ None of the CV models are used in either case. The model trained on the full training set is used, and predictions are made either on the validation_frame (for valid=T) or training_frame (for train=T). The numbers returned by h2o.mse() for example, when valid=T or train=T, are completely independent from the CV models. The only value that uses the CV models is h2o.mse(model, xval=T). $\endgroup$ – Erin LeDell Jan 3 '17 at 23:05
  • 1
    $\begingroup$ This is very useful information. Someone needs to put that into the help page. $\endgroup$ – horaceT Jan 3 '17 at 23:32
1
$\begingroup$

you can find more details from this in depth question about how H2O's cross validation works https://community.h2o.ai/questions/550/i-am-not-new-to-cross-validation-but-i-am-new-ish.html

and the following H2O docs sections: Cross Validation Section http://h2o-release.s3.amazonaws.com/h2o/master/3706/docs-website/h2o-docs/cross-validation.html

you can also look at the nfolds parameter explanation in the docs appendix

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.