18
$\begingroup$

My statistic professor says so, all the books that I look at state it: post-hoc-testing is unscientific. You must derive a hypothesis from theory first, and then collect data and analyse it.

But I really don't understand what the problem is.

Suppose, I see sales figures for different car colors and form the hypothesis that from numbers of different-colored cars sold the largest group of cars on the street shoud be white. So I sit at some street one day and note all the colors of all the cars that pass me. Then I do some tests and find whatever.

Now, suppose I was bored and sat at some street one day and noted all the colors of all the cars that passed me. Since I love graphs, I plot a pretty histogram and find that white cars form the largest group. So I think that maybe most cars on the street are white and perform some tests.

How and why do the results or the interpretation of the results of the post-hoc test differ from those of the theory-driven* hypothesis test?

* What's the name for the opposite of a post-hoc test, anyway?

I would like to add that most of our knowledge about the universe (the Earth moves around the Sun) is deduced post hoc from observation.

It seems to me that in physics it is perfectly okay to assume that it is not coincidence that the sun has been rising in the East for the last thousand years.

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ The issues are exemplified here & here. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Dec 22, 2014 at 10:39
  • $\begingroup$ @Scortchi Hmm, thank you, but all I can find is: " This would be an abuse of statistical testing, as has been amply explained and demonstrated in many places." The rest of the comments and answers seem not to explain the problem of post-hoc testing, but of testing in general. $\endgroup$
    – user14650
    Dec 22, 2014 at 10:57
  • 2
    $\begingroup$ Compare amoeba's answer (equivalent to your 1st scenario) to whuber's (equivalent to your 2nd). $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Dec 22, 2014 at 11:40
  • 3
    $\begingroup$ Just a note that the opposite of post-hoc is a priori. @whuber 's answer in the post linked above is pretty comprehensive, but you could look up exploratory data analysis vs. confirmatory data analysis. $\endgroup$
    – Peter Flom
    Dec 22, 2014 at 11:50
  • $\begingroup$ This is tangentially related but might be of interest to people reading this question: andrewgelman.com/2014/12/20/… $\endgroup$
    – shadowtalker
    Dec 22, 2014 at 20:11

6 Answers 6

Reset to default
14
$\begingroup$

"You know, the most amazing thing happened to me tonight. I was coming here, on the way to the lecture, and I came in through the parking lot. And you won't believe what happened. I saw a car with the license plate ARW 357. Can you imagine? Of all the millions of license plates in the state, what was the chance that I would see that particular one tonight? Amazing!" Richard Feynman

I feel that I am not in position to explain the deep technical aspects of this problem. However I think many of them can be reduced to an intuition.

In the first set up you start with some hypothesis which you verify on new data (from the designed experiment). Studying the sales figures can lead you to a very crafted well-designed experiment, where you really can decide how strong your answer should be (statistical power, p-values, sample size, and other many stuff).

In the second set up first of all is that you decide nothing about the strength of the answer. This is one problem. The second problem is that extracting the hypothesis from the same sample used for tests, will increase in a very uncontrollable way the chances that random patterns are interpreted as valuable information. What you do is to notice something (that white cars are in great number) and ask yourself if this is significant. The point is that you selected only a notable fact visible on that sample, discarding other hypotheses. Doing that you created favourable conditions for some hypothesis, and you break the assumptions of most apriori statistical tests.

It is not scientific to behave like you did not know about this leak, and pretend that it is an experiment with all its assumptions, when it is not true. It is scientific in this case to use post hoc analysis to formulate a hypothesis and design a brand new experiment in order to test it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But isn't an experiment, set up specifically for a hypothesis, the most extreme form of "favourable" conditions? $\endgroup$
    – user14650
    Dec 22, 2014 at 12:46
  • 2
    $\begingroup$ The only thing which "favours" an experiment is the solidity of the answer. And among other things, it tries to "not favour" a specific hypothesis. $\endgroup$
    – rapaio
    Dec 22, 2014 at 13:15
4
$\begingroup$

If you first collect data and then construct a theory based on the data, you are in danger of fitting a story to your observations. The problem is that we humans are extremely good at writing stories. Put another way: any data can be "explained" by a story, if the story is just convoluted enough.

This process provides for nice anecdotes. However, there is no reason why it should explain reality and/or provide good predictions. You need to set up and validate a model for that.

xkcd notes that this phenomenon pervades sports "commentary":

sports commentary

Related is the phenomenon of pareidolia: seeing patterns where none exist. See, for instance, the "Face" people saw in earlier satellite imagery of Mars:

Martian face

Plus, as you collect more data, you need to be careful you don't tweak your story in ever more bizarre ways to make it "continue" to "explain" your observations:

electoral precedent

Improve this answer
$\endgroup$
2
$\begingroup$

Science operates by forming hypotheses (which are of course are motivated by experience), making predictions based on those hypotheses and then testing them. Would it make sense to observe something in the past, generalize this observation into a theory, but then treat the past itself as a kind of retroactive experiment that automatically validates the theory? No, because the whole question was how well your theory generalizes, not whether or not it worked once in the past. This is why testing hypotheses suggested by the data is considered bad science.

Improve this answer
$\endgroup$
1
$\begingroup$

Your professor and the other answers are right that post-hoc analysis have problems. However, you are also right that a lot of good science comes from post-hoc analysis. The key point is that properly designed experiments should be preferred and that post-hoc analysis should be dealt with caution and with special tools to prevent missing spurious artefacts by actual discoveries. Wikipedia article on false discovery rate may give an insight on the problem.

Just to give a couple of examples:

  • If we take biometric measures on the whole world population of cattle we can conclude that cattle have two nostrils. That is in fact a post-hoc analysis, but most of biology, volcanology or history has been built this way. The reason we don't dismiss the fact that cattle have two nostrils is the evidence in favour of it being so overwhelming.
  • We take data from calves born in previous year in a given cattle farm. We realise that in every Tuesday under the full moon more than 50% of new born calves were female - except for public holidays in that country or winter Tuesdays. If we had previously made the hypothesis that those kinds of days produced more female calves, we could do an hypothesis test and accept (or reject) that hypothesis. However, if we take in account that it is just a post-hoc analysis, evidence won't be enough to reject an spurious phenomenon.

There is an often cited article that ironically dismisses all evidence of parachutes being useful as anecdotal - which is just a particularly bad class of evidence based on post-hoc analysis.

And to use a good example used by Stephan Kolassa's answer: a few dark spots resembling a face in Mars can be rejected as pareidolia, but something reproducing the Last Supper by Leonardo Da Vinci to the tinniest detail couldn't.

Improve this answer
$\endgroup$
0
$\begingroup$

If you don't have a theory backing your propositions, then even if your proposition is validated, it could be through coincidence and does not prove anything. For example, I find that I do potty when the sun rises and have been doing that for the past 10 years - based on this data, a post-hoc analysis tells me that there is a relationship between my doing potty and the sun rising, whereas what exists is merely a coincidence. The sun doesn't rise because you do potty or vice-versa.

Life is full of coincidences. Theory backed propositions eliminates such coincidences or pseudo-relationships.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If I have a theory and the results fit that theory, it could be coincidence just as well. Which is why theories cannot be validated, only falsified. And actually, there is a relationship between morning bowel movements and the sun rising, becausde the movements of the sun dictate the diurnal rhythm with in turn influences bowel movements. $\endgroup$
    – user14650
    Mar 16, 2015 at 7:50
0
$\begingroup$

Here is an intuition that you may find useful. If you are bored and count cars, you still have to remember that what you see is the result of some random process. In particular, the cars could have been different colors.

Therefore if you form the hypothesis that the most frequent color is white, if may be because it actually is but it could also be that the most frequent color is red but, on that particular experiment, the most frequent was white (which is always possible).

Now, if you do post-hoc, you will test for white being the most frequent and, given that the data suggested that very hypothesis, you may well conclude that white is the most frequent... At least, the data will never contradict the (post-hoc) hypothesis.

Improve this answer
$\endgroup$