0
$\begingroup$

I would like to compute prediction intervals for LightGBM at the sample level. In other words, given a certain row to be classified (supervised classification, not regression), what is the upper bound for my prediction, and the lower bound for my prediction, with a 90% certainty.

I have thought a bit about possible approaches:

  • Data density-based: ideally, this uncertainty would be linked to how likely it is to find this sample within the training dataset. There are a number of methods to estimate the underlying probability density function for a given sample - but I could not find a way to relate it with a confidence interval output. My only idea would be to use the density estimator to determine what is and isn't an outlier by establishing a 90-percentile range around the density scores - and then getting the performance variation for each group
  • GBM-specific methods: I have read some ideas about how this could be achieved - but none has satisfactory answers. They share doing some computation to the contributions of the trees, but this presumes that each tree is random and somewhat independent of the other (i.e. a random forest), which is not the case for LightGBM. I have two proposals, and I would like to know which sounds more sane
    • Bootstrapping the trees used to make a prediction: the final raw_score outputted by LightGBM is achieved by summing up the tree prediction of each individual tree that makes up the GBM. My assumption is that if we have a large number of trees, we could try sampling some (without replacement), sum their contributions and we will get a different score. We could do this a large number of times, and we will get a distribution of scores, which we can then get confidence intervals from.
    • Standard deviation of the trees: granted this one is the method presented in one the links, I would like to know your thoughts on whether it makes sense. Each tree can contribute 0 to the score, or a very large amount, so my expectation is to have a quite large standard dev (which will be even larger once we multiply it by 1.6 to get a 90% CI). Moreover, I'm not even sure if we could convert this raw score std to a final score std

I would really appreciate your input on this. To me, the bootstrapping method makes sense, but makes me a bit uneasy since we are randomly sampling trees that were not built randomly and complement each other (if we sample trees 0 1 2 and 4 its as if we are getting an output from a model that could never exist, since tree 4 would never be built without doing tree 3 first). OTOH, it sounds good.

Bonus question: how does one evaluate the correctness of the provided confidence intervals? How can I determine that one method of CI computation is better than another method?

Improve this question
$\endgroup$
5
  • $\begingroup$ What is the proper way to avoid cross-posting if the questions are flagged as out-of-topic in a certain stack exchange? $\endgroup$
    – Tiago Melo
    Commented Mar 20 at 8:27
  • $\begingroup$ After it's been closed/deleted, it's no problem trying another site. $\endgroup$
    – Ben Reiniger
    Commented Mar 20 at 12:03
  • $\begingroup$ As you recognise, the bootstrapping procedure won't work because the trees are not IID. I would suggest using conformal prediction for regression (Inductive conformal prediction - ICP) at the logit domain. It is pretty hammy but "fast". As a separate suggestion, we can bootstrap the whole sample, and use the OOB predicted scores to calculate PIs, our predictions in this case would be made by a model trained on a bootstrap sample. $\endgroup$
    – usεr11852
    Commented Mar 30 at 3:23
  • $\begingroup$ Right, but in the formulated question I'm dealing with classification, not regression - so ICP won't work (unless you are suggesting I treat the class probability estimation as a regression problem). Regarding the OOB method, I don't see how OOB predicted scores would be much different from sampling models, since OOB must presume some independence between the models (ie random forests)? $\endgroup$
    – Tiago Melo
    Commented Apr 3 at 12:01
  • $\begingroup$ 1. Almost correct, the suggestion is that the bounds are computed on the logit domain. Then we back-transform those to probabilities. Do remember that any boosting algorithm essentially does regression on its respective domain. 2. Bootstrapping the whole sample instead of the trees (as in your suggestion) would allow for no information leakage. Just bootstrap the whole sample and you are "safe" from that. (Sorry, I realised that OOB can be misinterpreted in the context of Random Forests, I mean Out-Of-Fold/Bag in the general sense.) $\endgroup$
    – usεr11852
    Commented Apr 3 at 23:23

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.