I'm quite new to Machine Learning, and after reading about the bias-variance tradeoff and overfitting/underfitting, several questions raised in my mind:
If I have a model with 15% error on train set and 14% error on validation set, and another model with 5% error on train set and 13% error on validation set, I understood it is generally better to pick the first model (note that 13% vs 14% is small but not negligible). Why is that? Does the low train error in the second model indicate it is more overfitted and thus would probably give worse results than the first on unseen data?
On the one hand, I read to avoid overfitting one needs to have, for example, lower tree depth or high value in the regulation parameter. But on the other hand, I read that to avoid overfitting one needs to optimize parameters on the validation set (and not the test set) and keep the train score just a little better than the test score (as in my first question). How do these two methods relate? Are they referring to the same type of overfitting? How do they both help reduce overfitting?
in this photo (taken from here) it is explained that for eta=0.1 (eta in this case is the learning rate in XGBOOST) the model is underfitting, and for eta=0.9 the model is overfitting. I can see how for eta=0.9 the model is overfitting (as the test error goes up) but for me it seems that also for eta=0.1 the model is overfitting - as the test error goes a little up while the train error goes down. What am I missing?
- Lets say I have 2 tree models. One one has
tree_depth=2and 10% error, and the other has
tree_depth=7and 9.7% error (so clearly the first model has lower complexity than the latter, but the second model is a better by a small but not negligible amount). I read it is recommended to pick the first model over the latter, but why is that? Is it because the first model is probably less overfitted? And if so, then why would I care if the complexity reduces error? Is it maybe because the first model would probably have less variance?
Note: answer to even a single question of those would be highly appreciated
Note 2: I have no statistics or math background so would highly appreciate (when possible) simple language