3
$\begingroup$

This is to double-check a few general ML principles. Could anyone please confirm:

Is the representation in the picture below "generally" correct?

enter image description here

So are the following implications correct?

1) real life performance is generally much worse than training performance, when I fit the on a small training set.

2) real life performance is very close to training performance, if the training set is sufficiently large.

3) Simpler models perform generally better than complex models, when the training is small, because they have less chances to overfit.

4) Complex model perform generally better than simpler models, when the training set is sufficiently large. Training set size mitigates overfitting.

5) The larger my training set, the more complex models I can afford to use. So the decision about what is the best model complexity, depends on how much data I have.

Thank you!

$\endgroup$
2
  • 1
    $\begingroup$ All true for "typical" ML applications, because usually you apply it to complex underlying processes. However, your complex model should not be more complex than the underlying process, otherwise you get no benefit from complexity, only the downside. For instance, if the underlying process is linear in reality, then modeling it as nonlinear only makes the model performance worse. $\endgroup$
    – Aksakal
    Sep 8 '17 at 19:43
  • $\begingroup$ @Aksakal back to the topic a year later :) I agree with your example, but also in that your example the general rule would apply: a sufficiently large training set would turn off the superfluous complexity of a model. Do you agree? $\endgroup$
    – elemolotiv
    Aug 19 '18 at 7:59
2
$\begingroup$

I think what you say is generally true for many machine learning problems of interest. For example, trying to categorize an image as x or y (something that is pretty difficult). In general, a larger training set means that you can increase model complexity.

1) real life performance is generally much worse than training performance, when I fit the on a small training set.

Generally, yes. It depends, though. The problem may be very easy, in which case you can separate all of the cases just based on a very small training set. For example, consider a perfectly separable training set where the test set will be identical. You could potentially learn a perfect model that generalizes perfectly from two examples only.

2) real life performance is very close to training performance, if the training set is sufficiently large.

As the training set size increases, the probability that the model approximates the true model increases. "Real life" examples are drawn from the true model. So, sure.

3) Simpler models perform generally better than complex models, when the training is small, because they have less chances to overfit.

Generally, maybe. If the decision boundary is nonlinear, a linear model will probably perform worse than a nonlinear model that overfits some. It is correct that a complex model will overfit more on a smaller training set.

4) Complex model perform generally better than simpler models, when the training set is sufficiently large.

True, if the task is not simple. If the problem is linearly separable, a simple model will perform as well as a complex model.

Training set size mitigates overfitting.

True

5) The larger my training set, the more complex models I can afford to use. So the decision about what is the best model complexity, depends on how much data I have.

True, but, again, it depends not just on how much but the nature of the data (and, for example, in the case of classification the boundary) as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.