Why not validate on the entire training set? We have a dataset with 10,000 manually labeled instances, and a classifier that was trained on all of this data. The classifier was then evaluated on ALL of this data to obtain a 95% success rate.
What exactly is wrong with this approach? Is it just that the statistic 95% is not very informative in this setup? Can there still be some value in this 95% number? While I understand that, theoretically, it is not a good idea, I don't have enough experience in this area to be sure by myself. Also note that I have neither built nor evaluated the classifier in question.
Common sense aside, could someone give me a very solid, authoritative reference, saying that this setup is somehow wrong? 
All I find on the Internet are toy examples supposed to convey some intuition. Here I have a project by professionals with an established track record, so I can't just say "this is wrong", especially since I don't know for sure.
For example, this page does say: 

Evaluating model performance with the data used for training is not acceptable in data mining because it can easily generate overoptimistic and overfitted models.

However, this is hardly an authoritative reference. In fact, this quote is plainly wrong, as the evaluation has nothing to do with generating overfitted models. It could generate overoptimistic data scientists who would choose the wrong models, but a particular evaluation strategy does not have anything to do with overfitting models per se. 
 A: If you validate on the entire training set, your ideal model is the one that just memorizes the data.  Nothing can beat it.
You say that "realistically this is not a model that just memorizes the data".  But why do you prefer other models?  This is the point of my reduction to absurdity of validating on all the data:  the main reason you don't like the model that memorizes everything it has seen is that it doesn't generalize at all.  What should it do given an input that it hasn't seen?  So you want a model that works in general rather than one that just works on what it has seen.  The way that you encode that desire for working well on unseen data is to set the validation data to be exactly that unseen data.
However, If you know that your training examples completely represent the true distribution, then go ahead and validate using them!
Also, contrary to the claims in your final paragraph, the quotation you cited is not "plainly wrong" and that "particular evaluation strategy" does have to do "with overfitting models".  Overfitting means fitting (the noise of) the provided training examples rather than the statistical relationships of general data.  By validating using seen data, you will prefer models that fit noise rather than those that work well using unseen data.
A: The argument is simple: when you build a model, you want this model to be efficient on NEW, UNSEEN data, right? Otherwise you don't need a model.
Then, your evaluation metric, let's say precision and recall, must give an idea of how well your model will behave on unseen data.
Now, if you evaluate on the same data that you used for training, your precision and recall will be biased (almost certainly, higher than they should), because your model has already seen the data. 
Suppose that you're a teacher writing an exam for some students. If you want to evaluate their skills, will you give them exercises that they have already seen, and that they still have on their desks, or new exercises, inspired by what they learned, but different from them?
That's why you always need to keep a totally unseen test set for evaluation. (You can also use cross-validation, but that's a different story).
A: @jpl has provided a good explanation of the ideas here.  If what you want is just a reference, I would use a solid, basic textbook.  Some well regarded books that cover the idea of cross-validation and why it's important might be:

*

*Harrell, F. (2010). Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis.  Springer.


*Hastie, T., Tibshirani, R., & Friedman, J. (2011). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer.
and/or


*James, G., Witten, T., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning: with Applications in R. Springer.
A: Here's my simple explanation.
When we model reality we want our models to be able not only to explain existing facts but also predict the new facts. So, the out-of-sample testing is to emulate this objective. We estimate (train) the model on some data (training set), then try to predict outside the training set and compare the predictions with the holdout sample.
Obviously, this is only an exercize in prediction, not the real prediction, because the holdout sample was in fact already observed. The real test in prediction happens only when you use the model on the data, which was not observed yet. For instance, you developed machine learning program for advertising. Only when you start using it in practice, and observe its performance you'll know for sure if it works or not.
However, despite the limitation of training/holdout approach, it's still informative. If your model only works in-sample, it's probably not a good model at all. So, this kind of testing helps weed out bad models.
Another thing to remember: let's say you conducted training/holdout sample validation of the model. However, when you want to use the model you probably will estimate the model on the entire dataset. In this case how applicable are the results of the out-of-sample validation of the model which was estimated on the training sample?
A: Others have answered your earlier paragraphs, so let me address your last one. Your point's validity depends on the interpretation of "evaluation". If it's used in the sense of a final run on unseen data to give a sense of how well your chosen model might be expected to work in the future, your point is correct.
If "evaluation" is used more in the sense of what I'd call a "test" set -- that is, to evaluate the results of training multiple models in order to choose one -- then evaluating on the training data will lead to overfitting.
A: All the other answers (especially related to over-fitting) are very good, but I would just add one thing.  The very nature of learning algorithms is that training them ensures they learn "something" common about the data they are exposed to. However, what we cannot be directly sure of, is exactly which features about the training data they end up actually learning.  As an example, with image recognition, it's very hard to be sure whether a trained neural network has learned what a face looks like, or something else that's inherent in the images.  An ANN could have just memorized what the shirts or shoulders or hair look like, for example.
That said, using a separate set of testing data (unseen by training) is one way to increase the confidence that you have a model that can be counted on to perform as expected with real-world/unseen data.  Increasing the number of samples and feature variability also helps.  What is meant by feature variability, is that you want to train with data that has as many variations which still count on each sample as possible. 
For example, with face data again, you want to show each particular face on as many different backgrounds as possible, and with as many variations in clothing, lighting, hair color, camera angles etc as possible.  This will help ensure that when the ANN says "face" it's really a face, and not a blank wall in the background that triggered the response.
A: Hastie et al have a good example in the context of cross-validation that I think also applies here. Consider prediction with an extremely high number of predictors on data where the predictors and outcomes are all independently distributed. For the sake of argument suppose that everything is Bernoulli with p = 0.5.
If you have enough variables then you'll have a few predictors that let you predict the outcomes perfectly. But, on new data, there's no way that you're going to get perfect accuracy.
This isn't exactly the same as your case but it does show an example where your method can really lead you astray.
