I'm doing a homework on Decision Trees, and one of the questions I have to answer is "Why are estimators built out of trees biased, and how does bagging help reduce their variance?".
Now, I know that overfitted models tend to have really low bias, because they try to fit all the data points. And, I had a script in Python that fitted a tree to some dataset (with a single feature. It was just a sinusoid, with some off points, picture below). So, I wondered "well, if I reeeeally overfit the data, can I get the bias to zero?". And, it turned out that, even with a depth of 10000, there are still some points through which the curve doesn't pass.
I tried searching for why, but I couldn't really find an explanation. I'm guessing that there may be some trees that would perfectly go through all the points, and that the ones I got were just "bad luck". Or that maybe a different dataset could've given me an unbiased result (maybe a perfect sinusoid?). Or even that, maybe the cuts made at the beginning made it impossible for further cuts to fully separate all the points.
So, taking into consideration this dataset (since it might be different for others), my question is: is it possible to overfit a tree to the point where the bias goes to zero, or is there always gonna be some bias, even if really small? And if there's always at least some bias, why does that happen?
P.S. I don't know if it might be relevant, but I used the
sklearn to fit the model to the data.