1
$\begingroup$

My dataset is 6.3 million observations, with 150 features for each one. 25 000 of these observations are positive case and the rest is negative case, so about 1:250 class balance.

I've been training GBM models on it (6-fold CV), with 1000 trees, 10 leaves per tree and I've changed the minimum number of observations per leaf parameter. I've tried 1, 10, 50, 100 and 500 so far and the AUC does not seem to change much. AUC has actually gone up from .885 to .900, but that could be due to the random seed picked or whatever.

I feel like at this point I should be drawing some conclusions about something. I previously had a feeling that since there's just 4200 positive cases per fold, I'd need the minimum observations per leaf to be small or it wouldn't capture anything. But after these results that's apparently not the case, so my understanding of how GBM works is probably flawed.

It could be that my choice of number of trees or number of leaves is completely wrong though, but since it takes an hour to train each model, I haven't gotten to trying wide range of these parameters yet.

Could anyone offer their thoughts on this situation?

$\endgroup$
0
$\begingroup$

Learning more about boosted models (Elements of Statistical Learning) I've pinpointed where my understanding was faulty. At least I hope so :)

The minimum number of observations is "per split" and the model tries to use the best feature to make the best split between positive and negative classes. Since I have a huge number of obsevations, I can make many perfectly OK splits where both sides still end up with a large enough number of observations.

I'd imagine I would run into serious underfitting trouble if I went with a "tree depth" * "minimum observations per leaf" > 6.3 mln. I did try up to 10 (number of leaves) * 30 000 (minimum observations) and the precision and recall both started to decrease, while AUC stayed the same. So it seems like, at least on my dataset, that one should keep the minimum number of observations either not confined on both sides or just limit it to 10 or so to avoid overfitting on splits where very few observations are split away.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.