0
$\begingroup$

I am randomly holding out 10% of data out of the whole dataset as test.data and train the GBM model on a remaining 90% of rows train.data (with x and y provided, no nfolds or validation data set provided )...

Once I print it, the trained model - it shows auc measure of 1 or 0.999.

But when I actually validate predictions against the initially held out 10% of data the auc comes back to around 0.91

I think I ruled out data leak, reviewed the code and looked at the variables importance with legit variables being appropriately important.

What is going on?

$\endgroup$
2
$\begingroup$

If you only provide H2O's GBM with a training_frame, then the only metrics it can produce is training metrics. That's why your AUC is so high -- it's training AUC. You should not assume that you training metrics will be similar to your test set metrics. Use h2o.performance(model, test.data) to generate test set metrics. If you want AUC specifically, then use h2o.auc(h2o.performance(model, test.data)).

$\endgroup$
  • $\begingroup$ But what is it calculated against ? I mean the training AUC ? And why h2o is providing it ? is it not that it will always be close to 1 on any training data ? (if it is calculated the way i think) $\endgroup$ – suprvisr Apr 24 '17 at 22:40
  • $\begingroup$ Training AUC is calculating using the predictions on the training data (generated by the model trained on the same data), along with the training data labels. Training AUC will not always be close to 1, it depends on the problem. $\endgroup$ – Erin LeDell Apr 25 '17 at 0:11
  • $\begingroup$ H2O provides training metrics because some people find them useful. However, to get an estimate of your generalization error, you should use test set metrics instead. $\endgroup$ – Erin LeDell Apr 25 '17 at 0:11
  • $\begingroup$ For my teaching, can you give me example, when AUC evaluated on training data (same data that model is trained) will return AUC less than 1 ? $\endgroup$ – suprvisr Apr 25 '17 at 0:12
  • $\begingroup$ Sorry, that's outside the scope of this question. You should be able to create an example of this using the dataset posted in the example code here: docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/… $\endgroup$ – Erin LeDell Apr 25 '17 at 0:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.