I'm curious to know whether we have such a case and if they exist how good/bad it is! I'm coming from this question and I'm thinking whether overfitting is an unpreventable concept that occurs regardless of what we do and whether it is bad or not! based on the answer to this question.
$\begingroup$
$\endgroup$
3
-
5$\begingroup$ Imagine that there is a relationship $y=a+bx $, I.e. your data is noise free. Then you will estimate the correct relationship after observing two (distinct) (y_i,x_i). $\endgroup$– SebastianCommented Oct 6, 2019 at 8:57
-
$\begingroup$ in that case, would we sill be saying, overfitting has occured? cause I guess overfitting refers to fitting to the noise in the input data in addition to the actual features and thus by definition this is contradictory to the term.is it not? $\endgroup$– HosseinCommented Oct 6, 2019 at 18:55
-
$\begingroup$ In a classification problem, it is quite possible to 100% accuracy test and training if the classes are separable. For example with the classic iris data set, most half-decent ML methods can get the setosa/not-setosa completely correct: I provided an example doing this for another question - you may have already seen this $\endgroup$– HenryCommented Oct 6, 2019 at 20:05
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
100% accuracy can clearly be achieved (also on the validation or unknown data) for some problem settings, but I guess those are rare cases. At the latest when the influence of the random noise "blurs" the borders of the data enough, the accuracy most likely will go down on unknown data, while it still may be up on the training data due to overfitting.
An example of a case where 100% accuracy is possible are one of the first experiments with neural nets, where researchers built AND/OR/XOR gates using neural nets. Back in these days I think, they didn't train the nets for these operations, but you surely could do that. The result will be 100% accuracy.
But this is a very special case. In fact, you probably would train the neural net with all the inputs it could ever see. Maybe you could even skip some of the inputs in your training set and it would still reconstruct the logical operation but that is quite limited. So in this case you don't really have a split between training and validation data because you train on the whole space of possible inputs. Of course such settings are not the typical settings for the application of machine learning. Why would you bother to train a ML model if you already labelled / can label the whole space of possible input by another method?
-
2$\begingroup$ Even if you've collected and labeled the entire possible input space, training an ML model can be worthwhile: there's a good chance that the model will be smaller than the input set. $\endgroup$– MarkCommented Oct 6, 2019 at 19:18
-
$\begingroup$ You're right. E.g. in case of a regression problem. $\endgroup$– jottbeCommented Oct 6, 2019 at 20:10