I'm quite new to Machine Learning, and after reading about the bias-variance tradeoff and overfitting/underfitting, several questions raised in my mind:

  1. If I have a model with 15% error on train set and 14% error on validation set, and another model with 5% error on train set and 13% error on validation set, I understood it is generally better to pick the first model (note that 13% vs 14% is small but not negligible). Why is that? Does the low train error in the second model indicate it is more overfitted and thus would probably give worse results than the first on unseen data?

  2. On the one hand, I read to avoid overfitting one needs to have, for example, lower tree depth or high value in the regulation parameter. But on the other hand, I read that to avoid overfitting one needs to optimize parameters on the validation set (and not the test set) and keep the train score just a little better than the test score (as in my first question). How do these two methods relate? Are they referring to the same type of overfitting? How do they both help reduce overfitting?

  3. in this photo enter image description here (taken from here) it is explained that for eta=0.1 (eta in this case is the learning rate in XGBOOST) the model is underfitting, and for eta=0.9 the model is overfitting. I can see how for eta=0.9 the model is overfitting (as the test error goes up) but for me it seems that also for eta=0.1 the model is overfitting - as the test error goes a little up while the train error goes down. What am I missing?

  1. Lets say I have 2 tree models. One one has tree_depth=2 and 10% error, and the other has tree_depth=7 and 9.7% error (so clearly the first model has lower complexity than the latter, but the second model is a better by a small but not negligible amount). I read it is recommended to pick the first model over the latter, but why is that? Is it because the first model is probably less overfitted? And if so, then why would I care if the complexity reduces error? Is it maybe because the first model would probably have less variance?

Note: answer to even a single question of those would be highly appreciated

Note 2: I have no statistics or math background so would highly appreciate (when possible) simple language

  • 2
    $\begingroup$ Good questions (+1) - but bundling several questions together isn't usually a good idea. The first three are quite closely related; but the last, at least should be split out into a separate CV question, & probably amplified a little. (The third could also do with some clarification - what's "eta"? - iterations of what?) $\endgroup$ Sep 8 '17 at 11:31
  • 1
    $\begingroup$ Very well, I'll remove the 4th question for now and make a new post later (note that I added another major misunderstanding of mine as the 4th question, though I think this one relates to the other 3) $\endgroup$
    – dan
    Sep 8 '17 at 11:34
  • $\begingroup$ On #2: "regulation parameter"? $\endgroup$ Aug 10 '20 at 11:34
  • $\begingroup$ Have you actually used validation? I see "training" data and "testing" data, though it looks as if you have used the testing data for validation. My view of best practice and language is to separate out test data at the very beginning and use it once at the very end on your final model. You can then use part of the training data for (cross-)validation to select and tune the model and hyperparameters $\endgroup$
    – Henry
    Sep 16 '20 at 7:18

As for number 3:

Model with eta = 0.9 clearly shows overfitting, as the more it is trained, the more it is able to predict the training set and the more it is unable to predict the testing set. I can only think model with eta = 0.1 is underfitting because compared to other eta values, it performs the worse. Not only it shows about the same test error trend with eta 0.5, but it also shows worse training error.

Note that the small perturbation in test error is quite normal (and can be caused by and treated as randomness) as long as it does not show increasing trend like in model with eta = 0.9

As for number 5:

The judgment whether reduction 0.3% in error is worth the cost of model complexity depends on practical use. If I were to choose, given some arbitrary context that I don't really know about, I'd pick the first because personally I think reduction of 0.3% in error does not justify adding 5 more depth to the tree and therefore creating unstable prediction. Of course, "negligible" varies from one application to another.

  • $\begingroup$ Does $\eta=0.1$ actually perform the worst on the testing data? My eyes suggest it may just about be the best of the three with the red lines $\endgroup$
    – Henry
    Sep 16 '20 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.