Why can't we mix exploratory model building and statistical tests for validating a known model? This short article attempts to expand on a lesson that was drilled in to me during my stats courses. That is, the point that some sorts of tests should never be mixed with ANOVA. It even contains this famous quote:

"It is "well known" to be "logically unsound and practically misleading" to make inference as if a model is known to be true when it has, in fact, been selected from the same data to be used for estimation purposes."

After reading the content of the link, I've came away more confused than I was when I started. Why can't we mix these two approaches? What examples do we have of two tests that give misleading results when used together?
 A: The basic problem here is that, if you use the data to identify which model and hypotheses you want to test, then the latter is dependent upon the the data.  Generally speaking, what happens is that people use exploratory data analysis to identify a "pattern" that appears to exist in the data, and then they formulate a model to test whether this pattern is due merely to randomness.  However, the very fact that they chose their test based on observation of the pattern in the exploratory phase biases the test (heavily) in favour of confirmation of that pattern.  If you are careful, it is possible to use adjustment for multiple comparison to correct for this, but often that is not done.
Philosophy Professor Ronald Giere explained the matter in more general terms in his book on scientific reasoning:

“If the known facts were used in constructing the model and were thus built into the resulting hypothesis … then the fit between these facts and the hypothesis provides no evidence that the hypothesis is true [since] these facts had no chance of refuting the hypothesis.”
——— Giere, R.N. (1984) Understanding Scientific Reasoning, p. 408.

The best procedure to avoid this kind of confirmation bias is to separate the available data into parts (via random sampling) and use part of the data for exploratory analysis and the other part of the data for confirmatory analysis.  This ensures that any "patterns" identified in exploratory analysis are tested formally using separate data that has not had an effect on the choice of model/hypothesis.


An example: Appealing to the infinite monkey theorem, suppose you put one-million monkeys at one-million typewriters and have them each bang out a string of characters.  You then look at the character strings to see if any of them bear any resemblance to a syntactically correct sentence.  You find that one of your monkeys ---let's call him Hector--- typed the following string:
$$\text{pegs arre legs]}$$
That is pretty extraordinary, and it is very close to a syntactically-correct English sentence, though there is failure to capitalise the first letter, a spelling mistake, and there is a ] instead of a full-stop (still pretty amazing for a monkey).  Now, suppose that on the basis of this exploratory phase, you formulate the hypothesis that Hector knows (roughly) how to type in English (subject to a few errors).  If you were to use the above character string as the data for your test, you would get an extremely low p-value and this would confirm that there is extremely strong evidence for this hypothesis.  Thus, you would conclude that Hector knows (roughly) how to type in English.  Does that conclusion seem correct to you?
Now, the infinite monkey theorem says that if you have enough monkeys, you are eventually going to get one that writes a syntactically-correct sentence.  So, if a million monkeys isn't enough, you might use a billion, etc.  Using the above procedure, you are essentially guaranteed to conclude that one of the monkeys knows how to type in English.  The problem here is that you have selected Hector for testing, out of a large number of monkeys, precisely because he wrote a character string that is close to a syntactically-correct sentence.  In short, you essentially chose this monkey for testing precisely because the p-value for his test is the lowest of all of them.  If you then use that same sentence as the evidence in the test, it is guaranteed to give you a low p-value because that is how you selected it.
Now, in this case, there is a pretty simple fix for this problem.  If you want to test whether Hector can type in English, you sit him down at the typewriter again and have him type another character string.  This time he types:
$$\text{wek[[jg   ooodd2}$$
This is nowhere near a syntactically-correct English sentence, and it essentially has a p-value near one.  So, Hector doesn't know how to type English.  The only reason we say something close to a syntactically-correct sentence the first time was that we tested a large number of monkeys and then selected the one that gave the most extraordinary outcome.

A: I think a classic example is stepwise regression, where you use the data to choose variables to build your model based on some criterion, such as F tests or t tests, and then use that model to perform inference. The key point is that hypothesis testing assumes you specify the model a priori and then gather data and test the hypothesis, but here you are using the data to build your model, and then using the same data to fit the model. Were you to neglect the fact that you used the data to build the model, and then ran the model on the same data and calculated p-values and confidence intervals for those variables, you'd be falling into the trap that quote is warning you against. Importantly, this matters if your goal is to do something other than just improve prediction within the data you collected, which is typically the case for any analysis (similar to how if you only care about the sample results, you don't need to do hypothesis testing--the observed values are the true values of that specific sample).
The main problem is that building and then testing a model using the same data fails to treat the observed patterns as random variables themselves. When you build a model based on having realized properties, it may be because of their underlying true property or because of noise, but conditional on selecting it, when you later test it, it will by construction have those properties because that's how you selected it. At the extreme, suppose you build your model based on whether the variables are statistically significant: then when you test that model on the same data, of course all the variables are going to be significant, but that's because they only enter your model conditional on being significant (based on the same data).
