7
$\begingroup$

We have a dataset with 10,000 manually labeled instances, and a classifier that was trained on all of this data. The classifier was then evaluated on ALL of this data to obtain a 95% success rate.

What exactly is wrong with this approach? Is it just that the statistic 95% is not very informative in this setup? Can there still be some value in this 95% number? While I understand that, theoretically, it is not a good idea, I don't have enough experience in this area to be sure by myself. Also note that I have neither built nor evaluated the classifier in question.

Common sense aside, could someone give me a very solid, authoritative reference, saying that this setup is somehow wrong?

All I find on the Internet are toy examples supposed to convey some intuition. Here I have a project by professionals with an established track record, so I can't just say "this is wrong", especially since I don't know for sure.

For example, this page does say:

Evaluating model performance with the data used for training is not acceptable in data mining because it can easily generate overoptimistic and overfitted models.

However, this is hardly an authoritative reference. In fact, this quote is plainly wrong, as the evaluation has nothing to do with generating overfitted models. It could generate overoptimistic data scientists who would choose the wrong models, but a particular evaluation strategy does not have anything to do with overfitting models per se.

$\endgroup$
  • 4
    $\begingroup$ A lot of people (rightly or wrongly) use the model evaluations to select their final model. If so, there is a sense in which a particular (inappropriate) evaluation strategy can generate overfitted [final] models. $\endgroup$ – gung Jan 2 '15 at 16:44
  • 2
    $\begingroup$ I would simply raise your doubts more directly. Given that you say that they are professionals with an established track record, why don't you specifically ask them why they did not use an independent test set. I suspect it is a misunderstanding - maybe they used a separate test set and then subsequently trained on all the data. $\endgroup$ – seanv507 Jan 2 '15 at 18:58
  • $\begingroup$ @seanv507, we are going to ask. I am just making sure there is no cutting-edge research in the area of validation on the training set... $\endgroup$ – osa Jan 2 '15 at 22:16
  • $\begingroup$ Whilst not a complete answer to your question, perhaps some intuition can be gained from considering the following case. If an algorithm explicitly memorised all 10,000 of your labelled instances, it would have 100% accuracy when shown any of those instances. But what would it do with a label that it had never seen before .... $\endgroup$ – image_doctor Jan 3 '15 at 1:41
  • $\begingroup$ Now that I think about it, by the nature of learning, whatever it is, any sensible model should do better when it has seen the data it is being tested on. It would be really strange if a model did better on other real-world data sets than on the one it was trained on, and it seems statistically unlikely to get the same, say, 81% in both cases. $\endgroup$ – osa Jan 3 '15 at 3:51
11
$\begingroup$

@jpl has provided a good explanation of the ideas here. If what you want is just a reference, I would use a solid, basic textbook. Some well regarded books that cover the idea of cross-validation and why it's important might be:

$\endgroup$
22
$\begingroup$

The argument is simple: when you build a model, you want this model to be efficient on NEW, UNSEEN data, right? Otherwise you don't need a model.

Then, your evaluation metric, let's say precision and recall, must give an idea of how well your model will behave on unseen data.

Now, if you evaluate on the same data that you used for training, your precision and recall will be biased (almost certainly, higher than they should), because your model has already seen the data.

Suppose that you're a teacher writing an exam for some students. If you want to evaluate their skills, will you give them exercises that they have already seen, and that they still have on their desks, or new exercises, inspired by what they learned, but different from them?

That's why you always need to keep a totally unseen test set for evaluation. (You can also use cross-validation, but that's a different story).

$\endgroup$
  • 1
    $\begingroup$ Yes, thank you, I do know all of that. Is there any evidence to back it up, something that I can point to and say "look, here those 10 articles in JEEE and this video all say that this 95% number is totally meaningless"? $\endgroup$ – osa Jan 2 '15 at 17:14
  • $\begingroup$ Well, you could just explain them what common sense dictates. I think it would be quite difficult to find a scientific article going back to these basics, but you can check on any supervised learning course, for example this one, picked at random on google: isys.ucl.ac.be/etudes/cours/linf2275/04classification.pdf (from slide 68) $\endgroup$ – jpl Jan 2 '15 at 17:38
  • 2
    $\begingroup$ If you want to convince someone, you can train one model that achieves 0 error on your data using a big neural net, k-nearest neighbour, an SVM or a random forest. (Or a table that memorises that data.) It should be clear, that this is not a property that will also hold for future data. $\endgroup$ – bayerj Jan 2 '15 at 18:15
  • 2
    $\begingroup$ Overfitting is certainly one of the results of this: overfitting refers to constructing a model that fits your available data perfectly but is too specific to be likely of use for general prediction of new data. That's exactly what this is. It's a different kind of overfitting, in a way, from the kind that is due to models with overly high degree polynomials or similar, but it refers to the same problem. $\endgroup$ – Joe Jan 2 '15 at 20:04
  • 1
    $\begingroup$ Right, I edited my answer and removed my last comment about overfitting. $\endgroup$ – jpl Jan 3 '15 at 15:01
5
$\begingroup$

If you validate on the entire training set, your ideal model is the one that just memorizes the data. Nothing can beat it.

You say that "realistically this is not a model that just memorizes the data". But why do you prefer other models? This is the point of my reduction to absurdity of validating on all the data: the main reason you don't like the model that memorizes everything it has seen is that it doesn't generalize at all. What should it do given an input that it hasn't seen? So you want a model that works in general rather than one that just works on what it has seen. The way that you encode that desire for working well on unseen data is to set the validation data to be exactly that unseen data.

However, If you know that your training examples completely represent the true distribution, then go ahead and validate using them!

Also, contrary to the claims in your final paragraph, the quotation you cited is not "plainly wrong" and that "particular evaluation strategy" does have to do "with overfitting models". Overfitting means fitting (the noise of) the provided training examples rather than the statistical relationships of general data. By validating using seen data, you will prefer models that fit noise rather than those that work well using unseen data.

$\endgroup$
  • $\begingroup$ Yes, but realistically this is not a model that just memorizes the data. This is some kind of a standard classifier, I assume. It may actually be a very good model, I just don't know it for sure. $\endgroup$ – osa Jan 2 '15 at 22:20
  • 3
    $\begingroup$ Even some "standard" classifiers memorize the data--or parts of it. $k$-Nearest neighbors obviously does that (and nothing more); indeed, with $k=1$, something has to be horribly wrong with your implementation if you fail to get 100% on the training set. Naive Bayes doesn't memorize all of the data, but it does extract a huge set of conditional probabilities. These really should be treated as sampled estimates with some uncertainty, but validating on the whole training set "promotes" these to exact population value. $\endgroup$ – Matt Krause Jan 3 '15 at 1:32
  • $\begingroup$ @MattKrause, THANK YOU! I see. So almost any binary/numeric classifier with one tunable parameter can be viewed as an interpolation, of sorts, with higher settings of the parameter leading to better fit on the set on which it is trained. Similarly, a classifier with multiple tunable parameters can be viewed as an optimization problem on the training set, since the only thing we can do is to optimize. (Unless, say, we tweak the parameters to get 50% worse fit on the training set than the maximum fit on the training set). So most classifiers are interpolators/optimizers in some space. $\endgroup$ – osa Jan 3 '15 at 3:40
  • $\begingroup$ @Neil G, I see, so the argument why cross-validation is not perfect is that the data comes "from the same distribution", and so measuring percent of correct guesses, while demonstrating that the model predicts something, may not represent the practical usability of the model in the context, in which we want to use it. $\endgroup$ – osa Jan 3 '15 at 3:44
  • $\begingroup$ @osa: cross-validation separates (many times) the data into a training set and a validation set. It doesn't not reuse training data for validation. I've edit my answer in response to your first comment. $\endgroup$ – Neil G Jan 3 '15 at 21:41
1
$\begingroup$

Here's my simple explanation.

When we model reality we want our models to be able not only to explain existing facts but also predict the new facts. So, the out-of-sample testing is to emulate this objective. We estimate (train) the model on some data (training set), then try to predict outside the training set and compare the predictions with the holdout sample.

Obviously, this is only an exercize in prediction, not the real prediction, because the holdout sample was in fact already observed. The real test in prediction happens only when you use the model on the data, which was not observed yet. For instance, you developed machine learning program for advertising. Only when you start using it in practice, and observe its performance you'll know for sure if it works or not.

However, despite the limitation of training/holdout approach, it's still informative. If your model only works in-sample, it's probably not a good model at all. So, this kind of testing helps weed out bad models.

Another thing to remember: let's say you conducted training/holdout sample validation of the model. However, when you want to use the model you probably will estimate the model on the entire dataset. In this case how applicable are the results of the out-of-sample validation of the model which was estimated on the training sample?

$\endgroup$
  • $\begingroup$ "Obviously, this is only an exercize in prediction, not the real prediction, because the holdout sample was in fact already observed." --- this is not so obvious, by the way, as the model did not observe this data. $\endgroup$ – osa Jan 2 '15 at 22:21
  • 1
    $\begingroup$ @osa, right but the modeler had the data. A modeler may have chosen the holdout sample consciously or subconsciously to prop up his out-of-sample performance metrics etc. That's why this kind of testing is not a true "back-testing", where the new data is truly new, wasn't available to a modeler at the time of modelling. $\endgroup$ – Aksakal Jan 2 '15 at 22:24
  • $\begingroup$ @Alsakal, good point about subconscious bias. I see, it's a similar kind of bias to that of reporting good statistics and ignoring bad ones. $\endgroup$ – osa Jan 3 '15 at 3:47
1
$\begingroup$

Others have answered your earlier paragraphs, so let me address your last one. Your point's validity depends on the interpretation of "evaluation". If it's used in the sense of a final run on unseen data to give a sense of how well your chosen model might be expected to work in the future, your point is correct.

If "evaluation" is used more in the sense of what I'd call a "test" set -- that is, to evaluate the results of training multiple models in order to choose one -- then evaluating on the training data will lead to overfitting.

$\endgroup$
1
$\begingroup$

All the other answers (especially related to over-fitting) are very good, but I would just add one thing. The very nature of learning algorithms is that training them ensures they learn "something" common about the data they are exposed to. However, what we cannot be directly sure of, is exactly which features about the training data they end up actually learning. As an example, with image recognition, it's very hard to be sure whether a trained neural network has learned what a face looks like, or something else that's inherent in the images. An ANN could have just memorized what the shirts or shoulders or hair look like, for example.

That said, using a separate set of testing data (unseen by training) is one way to increase the confidence that you have a model that can be counted on to perform as expected with real-world/unseen data. Increasing the number of samples and feature variability also helps. What is meant by feature variability, is that you want to train with data that has as many variations which still count on each sample as possible.

For example, with face data again, you want to show each particular face on as many different backgrounds as possible, and with as many variations in clothing, lighting, hair color, camera angles etc as possible. This will help ensure that when the ANN says "face" it's really a face, and not a blank wall in the background that triggered the response.

$\endgroup$
0
$\begingroup$

Hastie et al have a good example in the context of cross-validation that I think also applies here. Consider prediction with an extremely high number of predictors on data where the predictors and outcomes are all independently distributed. For the sake of argument suppose that everything is Bernoulli with p = 0.5.

If you have enough variables then you'll have a few predictors that let you predict the outcomes perfectly. But, on new data, there's no way that you're going to get perfect accuracy.

This isn't exactly the same as your case but it does show an example where your method can really lead you astray.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.