I have a fact as follows:
If we have a classifier trained on less training data is more likely to overfit.
This Fact is true.
I have a challenging question here. Is there any guarantee that when we have less training data we always get stuck in overfitting?
There are some trivial answers to consider that may serve as corner cases:
If you have sufficiently few samples (if need be, 0) your model will not only be unstable but even become mathematically impossible to train. You can quite reliably construct a situation where training will be unstable by taking just enough cases to make fitting mathematically possible.
In that sense, yes, you have a guarantee to get stuck in overfitting.
Overfitting is not something that happens on or off, it comes in degrees. And there are measures we can take against it. A certain amount of less training data may lead to no practical consequences. Plus, there are techniques to cope with less training data and instability (to some extent). So a given amount of less training samples does not guarantee overfitting.
A side question here is: what degree of coping do we still consider the same model? Most people would consider a PLS model that is adapted by using a few less latent variables "the same", but maybe not moving from single model to bagging.
For a given "perfect" model you could construct a very small data set that leads to the training producing those perfect parameters. Even though the training procedure would be highly unstable (small changes in the data set lead to large changes in the fit model), that particular model itself is not overfit.
So no, there is no guarantee that the unstable/overfitting training procedure will actually lead to a model that is overfit.
