How do bias, variance and overfitting relate to each other? [closed]

Question

I'm quite new to Machine Learning, and after reading about the bias-variance tradeoff and overfitting/underfitting, several questions raised in my mind:

1. If I have a model with 15% error on train set and 14% error on validation set, and another model with 5% error on train set and 13% error on validation set, I understood it is generally better to pick the first model (note that 13% vs 14% is small but not negligible). Why is that? Does the low train error in the second model indicate it is more overfitted and thus would probably give worse results than the first on unseen data?

2. On the one hand, I read to avoid overfitting one needs to have, for example, lower tree depth or high value in the regulation parameter. But on the other hand, I read that to avoid overfitting one needs to optimize parameters on the validation set (and not the test set) and keep the train score just a little better than the test score (as in my first question). How do these two methods relate? Are they referring to the same type of overfitting? How do they both help reduce overfitting?

3. in this photo (taken from here) it is explained that for eta=0.1 (eta in this case is the learning rate in XGBOOST) the model is underfitting, and for eta=0.9 the model is overfitting. I can see how for eta=0.9 the model is overfitting (as the test error goes up) but for me it seems that also for eta=0.1 the model is overfitting - as the test error goes a little up while the train error goes down. What am I missing?

5. Lets say I have 2 tree models. One one has tree_depth=2 and 10% error, and the other has tree_depth=7 and 9.7% error (so clearly the first model has lower complexity than the latter, but the second model is a better by a small but not negligible amount). I read it is recommended to pick the first model over the latter, but why is that? Is it because the first model is probably less overfitted? And if so, then why would I care if the complexity reduces error? Is it maybe because the first model would probably have less variance?

Note: answer to even a single question of those would be highly appreciated

Note 2: I have no statistics or math background so would highly appreciate (when possible) simple language

Answer 1

As for number 3:

Model with eta = 0.9 clearly shows overfitting, as the more it is trained, the more it is able to predict the training set and the more it is unable to predict the testing set. I can only think model with eta = 0.1 is underfitting because compared to other eta values, it performs the worse. Not only it shows about the same test error trend with eta 0.5, but it also shows worse training error.

Note that the small perturbation in test error is quite normal (and can be caused by and treated as randomness) as long as it does not show increasing trend like in model with eta = 0.9

---

As for number 5:

The judgment whether reduction 0.3% in error is worth the cost of model complexity depends on practical use. If I were to choose, given some arbitrary context that I don't really know about, I'd pick the first because personally I think reduction of 0.3% in error does not justify adding 5 more depth to the tree and therefore creating unstable prediction. Of course, "negligible" varies from one application to another.

Answer 2

In ML we want to follow Occam's razor. Prefer low complexity over high complexity, because we assume that we will perform better on new unseen data, if we have a less complex model (less overfitting). But there is of course also underfitting! So if you can show that a more complex model can outperform a less complex model on new data, then you should go with the more complicated model.

And to clarify bias, variance and overfitting: Bias is high if you have underfitted. You have a weak performance on train, test and validation sets. Variance is high if you have overfitted. Performance is good on training data and drops visibly on new unseen data.

Obviously this was a pretty quick summary. It always depends what a good and bad performance is, but I hope you got my point.

Answer 3

As for number 2

You are correct in your intuition in that the two concerns you raise elude to slightly different things.

In your first instance, the recommendation to choose a simpler model to avoid over-fitting, refers to the over-fitting of the models parameters - the things a model learns based on the training data. A more complex model will be more prone to 'memorising' the training data, thus lacking generalisability.

In your second instance, the recommendation to test parameters on the validation set refers to over-fitting hyper-parameters. These are the 'dials' you as a researcher can tweak to try to get the best out of the model. However, how do you know you haven't just optimised these to your test set by picking the one with the best test score? Add a validation set to see which model (i.e. set of hyper-parameters) does best, then check this against the test set. In your case the tree depth and regularisation parameters are both hyper-parameters. It's only needed if you are selecting from several models.

I found this discussion helpful when I was looking at this.