How to draw a conclusion about an hypothesis if one study obtains p < .05 and a second study obtains p > .05?

Question

I have a general question about comparing two studies:

Suppose study $A$ reports a $p$-value less than $0.05$. Also suppose study $B$ reports a $p$-value greater than $0.05$. Assume that both studies have the same $H_0$ and $H_a$. Both studies use different statistical techniques.

Which study is "correct"? Would you have to compare the statistical techniques used? If study $A$ was published before study $B$ would there be a "publication bias" in that most people would accept study $A$ as correct?

Answer 1

One thing to remember is that the p-value is a statistic---it takes your data and transforms it in a particular way. As a result, the p-value is a random variable. It has a mean and a variance. It is difficult to calculate the distribution of the p-value generally, but it is easy to calculate it under the null hypothesis: if the null hypothesis is true, the p-value has a uniform distribution from 0 to 1.

Since the p-value is random, it's possible that the two p-values that you are comparing are not statistically different. (Again, calculating p-values for differences in p-values is hard, typically requiring stronger assumptions than the problem gives. That two p-values are statistically the same, then, is hard to test.)

Andrew Gelman likes to say that "the difference between statistically significant and not is not statistically significant." In other words, 0.05 is not a magical cutoff and being below this value isn't necessarily different than being a little above due to the randomness of the p-value itself.

Another issue is that some procedures are more efficient than others. Typically, efficiency is gained by making additional assumptions about the data and using those assumptions to devise tweaks in the estimation procedure. It's possible that the two studies have the same estimate of the effect, but have different standard errors, leading to different p-values simply because one study makes additional assumptions that gets it lower standard errors.

Of real interest to you should be whether the two studies estimate the same effect. If the estimates are similar, most people would choose the more efficient/significant one unless they believe that the assumptions needed for the efficiency boost are not valid.

If the estimates of the effect itself are not similar, then you need to look at the two estimation procedures and determine which is more compelling---this isn't something that I can comment on without knowing the specifics.

Answer 2

Quite possibly BOTH studies are correct. Also possible that both are wrong. Or that either is correct.

What question did the two studies try to answer? What statistical techniques did they use? What were the parameter estimates for each? Did they use the same data? What data did each use?

Right now, your question is something like: One person says Scotch is better than beer. Another says it isn't. Who is right? They use different taste buds.

Answer 3

Assuming that both studies are testing the same null hypothesis, your question exposes an important problem with Neyman-Pearson type hypothesis testing. If we don't take the first result as final then we are not 'playing by the rules' and we don't control the rate of false positive results. That is why Fisher repeatedly complained that the N-P method was good for industrial quality control and acceptance procedures but all but useless for scientific enquiry.

If you really want a composite result from the two studies then you need to use the exact P values rather than the N-P binary division between significant and not significant. Google 'combining p values' to get a start (I'm on leave playing with a iPad so I don't have access to my resources, so that's the best I can do for you at the moment.)

Answer 4

There isn't really a credible "Wrong" and "Right" study based on p-values. You can run a shoddy study and get a really low p-value (indeed, its easy to do), or you can run run a study that is essentially a scientific masterpiece, and get a non-significant result. Or indeed you can run the same study multiple times, and have a spread of significant and non-significant results.

To answer some other bits of your question:

"Publication Bias": What you're asking about isn't actually publication bias, because you've implied both Study A and Study B are published (or at least findable). While there is evidence that a paper with a significant finding is more likely to be published, if they both are, what you're actually asking about is closer to "Citation Bias". And there's a distinct possibility that yes, people will favor the significant result, even if they really shouldn't.

Statistical Technique: YES. You should absolutely compare the two statistical techniques. There are two reasons for this:

1. One of them might be wrong/inappropriate. Scientists, even good ones, are fallible creatures. It's possible the method they used is inappropriate for their question or data - or that one study is pretty clearly superior. For example, if Study A doesn't bother adjusting for some really clear confounding, and Study B does, then yes, Study B probably "better", all other things being equal.
2. You can ascertain whether or not their results are actually conflicting. Some of the big canonical "Two studies disagree!" stories, especially in health, are often actually an issue of them asking very subtly different questions. These are often not clear at first blush, and often the study authors don't realize it at all. A good example would be Hernan's examination of the Women's Health Initiative findings about HRT versus the supposedly conflicting Nurses Health Study and other cohort studies. Turns out, a fair amount of it was about asking different questions (see: http://www.hsph.harvard.edu/faculty/miguel-hernan/files/hernan_biometrics05.pdf).

There is however a way to deal with all this. Meta-analysis is, in essence, a collection of techniques to synthesize the results of a literature, often in conflict (presumably you actually have more than two studies) and look at it as a coherent whole.