# How do bias, variance and overfitting relate to each other?

I'm quite new to Machine Learning, and after reading about the bias-variance tradeoff and overfitting/underfitting, several questions raised in my mind:

1. If I have a model with 15% error on train set and 14% error on validation set, and another model with 5% error on train set and 13% error on validation set, I understood it is generally better to pick the first model (note that 13% vs 14% is small but not negligible). Why is that? Does the low train error in the second model indicate it is more overfitted and thus would probably give worse results than the first on unseen data?

2. On the one hand, I read to avoid overfitting one needs to have, for example, lower tree depth or high value in the regulation parameter. But on the other hand, I read that to avoid overfitting one needs to optimize parameters on the validation set (and not the test set) and keep the train score just a little better than the test score (as in my first question). How do these two methods relate? Are they referring to the same type of overfitting? How do they both help reduce overfitting?

3. in this photo (taken from here) it is explained that for eta=0.1 (eta in this case is the learning rate in XGBOOST) the model is underfitting, and for eta=0.9 the model is overfitting. I can see how for eta=0.9 the model is overfitting (as the test error goes up) but for me it seems that also for eta=0.1 the model is overfitting - as the test error goes a little up while the train error goes down. What am I missing?

1. Lets say I have 2 tree models. One one has tree_depth=2 and 10% error, and the other has tree_depth=7 and 9.7% error (so clearly the first model has lower complexity than the latter, but the second model is a better by a small but not negligible amount). I read it is recommended to pick the first model over the latter, but why is that? Is it because the first model is probably less overfitted? And if so, then why would I care if the complexity reduces error? Is it maybe because the first model would probably have less variance?

Note: answer to even a single question of those would be highly appreciated

Note 2: I have no statistics or math background so would highly appreciate (when possible) simple language

• Good questions (+1) - but bundling several questions together isn't usually a good idea. The first three are quite closely related; but the last, at least should be split out into a separate CV question, & probably amplified a little. (The third could also do with some clarification - what's "eta"? - iterations of what?) – Scortchi - Reinstate Monica Sep 8 '17 at 11:31
• Very well, I'll remove the 4th question for now and make a new post later (note that I added another major misunderstanding of mine as the 4th question, though I think this one relates to the other 3) – dan Sep 8 '17 at 11:34
• On #2: "regulation parameter"? – Nuclear03020704 Aug 10 '20 at 11:34
• Have you actually used validation? I see "training" data and "testing" data, though it looks as if you have used the testing data for validation. My view of best practice and language is to separate out test data at the very beginning and use it once at the very end on your final model. You can then use part of the training data for (cross-)validation to select and tune the model and hyperparameters – Henry Sep 16 '20 at 7:18

• Does $\eta=0.1$ actually perform the worst on the testing data? My eyes suggest it may just about be the best of the three with the red lines – Henry Sep 16 '20 at 7:12