4
$\begingroup$

Generally, machine learning methods make little to no statistical assumptions. However, a key assumption they do make is that the data are IID.

What are the implications of a violation of the independence of the labels assumption for sequential gradient boosting? To be concrete, a Gradient Boosted Decision Tree.

In ensembles that exclusively use bagging (e.g., random forest), it is very easy to see what happens: the variance reduction from bagging results from the fact that the errors of the base learners are uncorrelated (say a white noise); the expected value of noise is zero and the noise diversifies away. If the independence assumption is violated in random forest, then you will see that the out-of-bag (OOB) error is unreasonably low (because the samples are correlated) and inconsistent with the validation error.

Is there a similar thought experiment one can do to make conclusions about what happens in GBDTs when the data have non-independent labels?

$\endgroup$
2
  • $\begingroup$ Did you ever find an answer to your question? Interested to know... $\endgroup$ Commented Jul 13, 2020 at 14:33
  • 1
    $\begingroup$ No formal answer. This seems like a big issue so I am surprised not to see something. $\endgroup$
    – Jonathan
    Commented Jul 21, 2020 at 15:08

0