# Why does a decision tree have low bias & high variance?

Questions

1. Does it depend on whether the tree is shallow or deep? Or can we say this irrespective of the depth/levels of the tree?
2. Why is bias low & variance high? Please explain intuitively and mathematically

A bit late to the party but i feel that this question could use answer with concrete examples.

I will write summary of this excellent article: bias-variance-trade-off, which helped me understand the topic.

The prediction error for any machine learning algorithm can be broken down into three parts:

• Bias Error
• Variance Error
• Irreducible Error

Irreducible error

As the name implies, is an error component that we cannot correct, regardless of algorithm and it's parameter selection. Irreducible error is due to complexities which are simply not captured in the training set. This could be attributes which we don't have in a learning set but they affect the mapping to outcome regardless.

Bias error

Bias error is due to our assumptions about target function. The more assumptions(restrictions) we make about target functions, the more bias we introduce. Models with high bias are less flexible because we have imposed more rules on the target functions.

Variance error

Variance error is variability of a target function's form with respect to different training sets. Models with small variance error will not change much if you replace couple of samples in training set. Models with high variance might be affected even with small changes in training set.

Consider simple linear regression:

Y=b0+b1x


Obviously, this is a fairly restrictive definition of a target function and therefore this model has a high bias.

On the other hand, due to low variance if you change couple of data samples, it's unlikely that this will cause major changes in the overall mapping the target function performs. On the other hand, algorithm such as k-nearest-neighbors have high variance and low bias. It's easy to imagine how different samples might affect K-N-N decision surface.

Generally, parametric algorithms have a high bias and low variance, and vice versa.

One of the challenges of machine learning is finding the right balance of bias error and variance error.

Decision tree

Now that we have these definitions in place, it's also straightforward to see that decision trees are example of model with low bias and high variance. The tree makes almost no assumptions about target function but it is highly susceptible to variance in data.

There are ensemble algorithms, such as bootstrapping aggregation and random forest, which aim to reduce variance at the small cost of bias in decision tree.

If the number of levels is too high i.e a complicated decision tree, the model tends to overfit.

Intuitively, it can be understood in this way. When there are too many decision nodes to go through before arriving at the result i.e number of nodes to traverse before reaching the leaf nodes is high, the conditions that you are checking against becomes multiplicative. That is, the computation becomes (condition 1)&&(condition 2)&&(condition 3)&&(condition 4)&&(condition5).

Only if all the conditions are satisfied, a decision is reached. As you can see, this will work very well for the training set as you are continuously narrowing down on the data. The tree becomes highly tuned to the data present in the training set.

But when a new data point is fed, even if one of the parameters deviates slightly, the condition will not be met and it will take the wrong branch.

Why does a decision tree have low bias & high variance? Does it depend on whether the tree is shallow or deep? Or can we say this irrespective of the depth/levels of the tree? Why is bias low & variance high? Please explain intuitively and mathematically.

Bias vs Variance

More Bias = error from the model being more simpler (does not fit the data very well)

More Variance = error from the model being more complex (fits the data too well, and learns the noise in addition to the inherent patterns in the data)

Everything is relative

I want to start by saying that everything is relative. Decision Tree in general has low bias and high variance that let's say random forests. Similarly, a shallower tree would have higher bias and lower variance that the same tree with higher depth.

Comparing variance of decision trees and random forests

Now with that ironed out, let's think why decision trees would be worse in variance (higher variance and lower bias) than let's say random forests. The way a decision tree algorithm works is that the data is split again and again as we go down in the tree, so the actual predictions would be made by fewer and fewer data points. Compared to that, random forests aggregate the decisions of multiple trees, and that too, less-correlated trees through randomization, hence the model generalizes better (=> performs more reliably across different datasets = lower variance). Similarly, we are making more simplifying assumptions on random forests to consult only a subset of data and features to fit a single tree, hence higher bias. BTW, similary, a tree with lower height = less reliant on fewer data points generalizes better and and has less variance compared to a deep tree.

1. A complicated decision tree (e.g. deep) has low bias and high variance. The bias-variance tradeoff does depend on the depth of the tree.

2. Decision tree is sensitive to where it splits and how it splits. Therefore, even small changes in input variable values might result in very different tree structure.

• I don't remember a single ordinary tree algorithm that's affected by scaling, they don't see the variables values, only the ranks. Commented Feb 19, 2017 at 15:44

Context: decision tree has low bias & high variance

Q. Does it depend on whether the tree is shallow or deep?

A. If the tree is shallow then we're not checking a lot of conditions/constrains ie the logic is simple or less complex, hence it automatically reduces over-fitting. This introduces more bias compared to deeper trees where we overfit the data. It can be imagined as we're deliberately not calculating more conditions means we're making some assumption (introduces bias) while creating the tree.

Q. Or can we say this irrespective of the depth/levels of the tree?

A. It can be thought of as more the depth means lesser the bias as we're relying on data more rather than assumptions.

Q. Why is bias low & variance high?