Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

It is well known that researchers should spend time observing and exploring existing data and research before forming a hypothesis and then collecting data to test that hypothesis (referring to null-hypothesis significance testing). Many basic statistics books warn that hypotheses must be formed a priori and can not be changed after data collection otherwise the methodology becomes invalid.

I understand that one reason why changing a hypothesis to fit observed data is problematic is because of the greater chance of committing a type I error due to spurious data, but my question is: is that the only reason or are there other fundamental problems with going on a fishing expedition?

As a bonus question, are there ways to go on fishing expeditions without exposing oneself to the potential pitfalls? For example, if you have enough data, could you generate hypotheses from half of the data and then use the other half to test them?

update

I appreciate the interest in my question, but the answers and comments are mostly aimed at what I thought I established as background information. I'm interested to know if there are other reasons why it's bad beyond the higher possibility of spurious results and if there are ways, such as splitting data first, of changing a hypothesis post hoc but avoiding the increase in Type I errors.

I've updated the title to hopefully reflect the thrust of my question.

Thanks, and sorry for the confusion!

share|improve this question
4  
1  
Taking another point of view on what has already been said: The essence of the scientific method is to make hypotheses and then try to falsify them for that they may become theories (if the falsification fails). Going on a fishing expedition is a valid way to find hypotheses that are worth falsifying in a later experiment, but you can never make and try to falsify a hypothesis in one go. In particular, if you are open to adjusting your hypothesis, you are not trying to falsify it anymore. Instead, when you adjust, you are falsifying your unadjusted hypothesis and form a new hypothesis. –  Wrzlprmft May 27 at 15:17
    
@jona, that's a great paper. I've already read papers by both Ioannidis and Schooler, but Simmons et al wonderfully illustrates the problem. –  post-hoc May 27 at 16:04
1  
I'm wondering whether you'll also find this paper relevant to your question: stat.columbia.edu/~gelman/research/published/multiple2f.pdf. It's not exactly on the same subject, but it addresses one aspect of it. –  a11msp May 27 at 16:22
1  
Data may cause you to change your hypothesis... but in that case you need to start gathering new data from scratch to confirm the new hypothesis. –  keshlam May 28 at 12:51
add comment

2 Answers 2

Certainly you can go on fishing expeditions, as long as you admit that it's a fishing expedition and treat it as such. A nicer name for such is "exploratory data analysis".

A better analogy might be shooting at a target:

You can shoot at a target and celebrate if you hit the bulls eye.

You can shoot without a target in order to test the properties of your gun.

But it's cheating to shoot at a wall and then paint a target around the bullet hole.

One way to avoid some of the problems with this is to do the exploration in a training data set and then test it on a separate "test" data set.

share|improve this answer
11  
It's hard to improve on Peter's answer. The unfortunate problem with much of data dredging is the lack of admission by the authors that the hypotheses were not fully pre-specified, i.e., not using the term 'exploratory'. Many, many researchers are dredging data to get a publishable paper and not following up with any attempt at validation (which would often disappoint them). –  Frank Harrell May 27 at 12:05
2  
Taking Frank Harrell's comment a step further: it's legitimate to explore some data and publish an intriguing finding... as an intriguing, exploratory finding that's subject to being reproduced/validated. The downside is: if someone else confirms your findings they may well get the glory, and if others do not confirm your results you were fooled by a spurious correlation. Bad if you have a big ego. Not to mention you would need to make your data and procedures publicly available, which many practitioners in many fields wont't do. And you should follow up with new data rather than moving on. –  Wayne May 27 at 13:26
8  
+1 But it's cheating to shoot at a wall and then paint a target around the bullet hole. –  WernerCD May 27 at 15:37
3  
@post-hoc well, it shouldn't raise eyebrows, but it might. Depends on whose eyes are under the brows! –  Peter Flom May 27 at 17:53
2  
Texas Sharpshooter Fallacy.. –  smci May 28 at 20:02
show 4 more comments

The problem with fishing expeditions is this: if you test enough hypotheses, one of them will be confirmed with a low p value. Let me give a concrete example.

Imagine you have are doing an epidemiological study. You have found 1000 patients that suffer from a rare condition. You want to know what they have in common. So you start testing - you want to see whether a particular characteristic is overrepresented in this sample. Initially you test for gender, race, certain pertinent family history (father died of heart disease before age 50, …) but eventually, as you are having trouble finding anything that "sticks", you start to add all kinds of other factors that just might relate to the disease:

  • is vegetarian
  • has traveled to Canada
  • finished college
  • is married
  • has children
  • has cats
  • has dogs
  • drinks at least 5 glasses of red wine per week

Now here is the thing. If I select enough "random" hypotheses, it starts to become likely that at least one of these will result in a p value less than 0.05 - because the very essence of p value is "the probability of being wrong to reject the null hypothesis when there is no effect". Put differently - on average, for every 20 bogus hypotheses you test, one of them will give you a p of < 0.05.

This is SO very well summarized in the XKCD cartoon http://xkcd.com/882/ :

enter image description here

The tragedy is that even if an individual author does not perform 20 different hypothesis tests on a sample in order to look for significance, there might be 19 other authors doing the same thing; and the one who "finds" a correlation now has an interesting paper to write, and one that is likely to get accepted for publication…

This leads to an unfortunate tendency for irreproducible findings. The best way to guard against this as an individual author is to set the bar higher. Instead of testing for the individual factor, ask yourself "if I test N hypotheses, what is the probability of coming up with at least one false positive". When you are really testing "fishing hypotheses" you could think about making a Bonferroni correction to guard against this - but people frequently don't.

There were some interesting papers by Dr Ioannides - profiled in the Atlantic Monthly specifically on this subject.

See also this earlier question with several insightful answers.

update to better respond to all aspects of your question:

If you are afraid you might be "fishing", but you really don't know what hypothesis to formulate, you could definitely split your data in "exploration", "replication" and "confirmation" sections. In principle this should limit your exposure to the risks outlined earlier: if you have a p value of 0.05 in the exploration data and you get a similar value in the replication and confirmation data, your risk of being wrong drops. A nice example of "doing it right" was shown in the British Medical Journal (a very respected publication with an Impact Factor of 17+)

Exploration and confirmation of factors associated with uncomplicated pregnancy in nulliparous women: prospective cohort study, Chappell et al

Here is the relevant paragraph:

We divided the dataset of 5628 women into three parts: an exploration dataset of two thirds of the women from Australia and New Zealand, chosen at random (n=2129); a local replication dataset of the remaining third of women from Australia and New Zealand (n=1067); and an external, geographically distinct confirmation dataset of 2432 European women from the United Kingdom and Republic of Ireland.

Going back a little bit in the literature, there is a good paper by Altman et al entitle "Prognosis and prognostic research: validating a prognostic model" which goes into a lot more depth, and suggests ways to make sure you don't fall into this error. The "main points" from the article:

Unvalidated models should not be used in clinical practice When validating a prognostic model, calibration and discrimination should be evaluated Validation should be done on a different data from that used to develop the model, preferably from patients in other centres Models may not perform well in practice because of deficiencies in the development methods or because the new sample is too different from the original

Note in particular the suggestion that validation be done (I paraphrase) with data from other sources - i.e. it is not enough to split your data arbitrarily into subsets, but you should do what you can to prove that "learning" on set from one set of experiments can be applied to data from a different set of experiments. That's a higher bar, but it further reduces the risk that a systematic bias in your setup creates "results" that cannot be independently verified.

It's a very important subject - thank you for asking the question!

share|improve this answer
7  
This brings to mind: xkcd.com/882 –  Jens May 27 at 13:48
2  
@jens - that is a far more eloquent explanation than the one I gave... Thanks for that link. As usual - do hover your mouse over the cartoon for a little zinger. –  Floris May 27 at 13:51
    
Ioannides and the Lehrer article was the path that brought me here. Your example is similar to the example in Simmons et al mentioned by @jona. It's a very good way of explaining the increase likelihood of Type I errors, but are there other reasons why it's bad? –  post-hoc May 27 at 16:08
1  
The problem with data dredging in general is that you risk confounding "correlation" with "causation". By coming up with a reasonable hypothesis first, then confirming that it helps explain the observations, you limit the risk of confusing the two. "Big Data" often goes the other way - their modus operandi is "if I analyze enough data I will see patterns that held true in the past and that will continue to hold in the future". Sometimes it works, sometimes it doesn't. Statistics should never become a substitute for thinking and understanding - only ever a confirmation. –  Floris May 27 at 16:26
5  
I don't think the primary issue is correlation vs. causation. It is easy to do lousy correlational analysis only to find that associations do not replicate. –  Frank Harrell May 27 at 18:57
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.