Assume that a model has 100% accuracy on the training data, but 70% accuracy on the test data. Is the following argument true about this model?
It is obvious that this is an overfitted model. The test accuracy can be enhanced by reducing the overfitting. But, this model can still be a useful model, since it has an acceptable accuracy for the test data.
I think the argument is correct. If 70% is acceptable in the particular application, then the model is useful even though it is overfitted (more generally, regardless of whether it is overfitted or not).
While balancing overfitting against underfitting concerns optimality (looking for an optimal solution), having satisfactory performance is about sufficiency (is the model performing well enough for the task?). A model can be sufficiently good without being optimal.
Edit: after the comments by Firebug and Matthew Drury under the OP, I will add that to judge whether the model is overfitted without knowing the validation performance can be problematic. Firebug suggests comparing the validation vs. the test performance to measure the amount of overfitting. Nevertheless, when the model delivers 100% accuracy on the training set without delivering 100% accuracy on the test set, it is an indicator of possible overfitting (especially so in the case of regression but not necessarily in classification).
set.seed(100) for an illustration like the phenomenon described here and set.seed(15) for the opposite. Perhaps better to say "an indicator of possible overfitting"
$\endgroup$
In my past project with Credit Card Fraud detection, we intentionally want to over fit the data / hard coded to remember fraud cases. (Note, overfitting one class is not exactly the general overfitting problem OP talked about.) Such system has relatively low false positives and satisfy our needs.
So, I would say, overfitted model can be useful for some cases.
Maybe: beware. When you say that 70% accuracy (however you measure it) is good enough for you, it feels like you're assuming that errors are randomly or evenly distributed.
But one of the ways of looking at overfitting is that it happens when a model technique allows (and its training process encourages) paying too much attention to quirks in the training set. Subjects in the general population that share these quirks may have highly-unbalanced results.
So perhaps you end up with a model that says all red dogs have cancer -- because of that particular quirk in your training data. Or that married people between the ages of 24 and 26 are nearly guaranteed to file fraudulent insurance claims. Your 70% accuracy leaves a lot of room for pockets of subjects to be 100% wrong because your model is overfit.
(Not being overfit isn't a guarantee that you won't have pockets of wrong predictions. In fact an under-fit model will have swaths of bad predictions, but with overfitting you know you are magnifying the effect of quirks in your training data.)
No they can be useful, but it depends on your purpose. Several things spring to mind:
Cost-Sensitive Classification: If your evaluation function overweights TPR and underweights FPR, we use $F_\beta$ score with $\beta \gg 1$. (such as @hxd1011's answer on antifraud)
Such a classifier can be really useful in an ensemble. We could have one classifier with normal weights, one that overweights TPR, one that overweights FNR. Then even simple rule-of-three voting, or averaging, will give better AUC than any single best classifier. If each model uses different hyperparameters (or subsampled training-sets, or model architectures), that buys the ensemble some immunity from overfitting.
Similarly, for real-time anti-spam, anti-fraud or credit-scoring, it's ok and desirable to use a hierarchy of classifiers. The level-1 classifiers should evaluate really fast (ms) and it's ok to have a high FPR; any mistakes they make will be caught by more accurate, fully-featured, slower higher-level classifiers or ultimately human reviewers. Obvious example: prevent fake-news headlines from Twitter account takeovers like the 2013 "White House bomb attack kills three" from affecting $billions of trading within ms of posting. It's ok for the level-1 classifier to flag that as positive for spam; let's allow it takes a little while to (automatically) determine the truth/falsehood of sensational-but-unverified news reports.
I'm not denying that an overfitted model could still be useful. But just keep in mind that this 70% could be a misleading information. What you need in order to judge if a model is useful or not is the out-of-sample error, not the testing error (the out-of-sample error is not known, so we have to estimate it using a blinded testing set), and that 70% is barely the good approximation.
In order to make sure that we're on the same page on the terminology after the comment of @RichardHardy, let's define the testing error as the error obtained when applying the model on the blind testing set. And the out-of-sample error is the error when applying the model to the whole population.
The approximation of the out-of-sample error depends on two things: the model itself and the data.
An "optimal" model yields to an (testing) accuracy that scarcely depends on the data, in this case, it would be a good approximation. "Regardless" of the data, the prediction error would be stable.
But, an overfitted model's accuracy is highly dependent of the data (as you mentioned 100% on the training set, and 70% on the other set). So it might happens that when applying to another data set, the accuracy could be somewhere lower than 70% (or higher), and we could have bad surprises. In other words, that 70% is telling you what you believe it to be, but it is not.
