I am randomly holding out 10% of data out of the whole dataset as test.data and train the GBM model on a remaining 90% of rows train.data (with x and y provided, no nfolds or validation data set provided )...
Once I print it, the trained model - it shows auc measure of 1 or 0.999.
But when I actually validate predictions against the initially held out 10% of data the auc comes back to around 0.91
I think I ruled out data leak, reviewed the code and looked at the variables importance with legit variables being appropriately important.
What is going on?