# How to draw a conclusion about an hypothesis if one study obtains p < .05 and a second study obtains p > .05?

I have a general question about comparing two studies:

Suppose study $A$ reports a $p$-value less than $0.05$. Also suppose study $B$ reports a $p$-value greater than $0.05$. Assume that both studies have the same $H_0$ and $H_a$. Both studies use different statistical techniques.

Which study is "correct"? Would you have to compare the statistical techniques used? If study $A$ was published before study $B$ would there be a "publication bias" in that most people would accept study $A$ as correct?

-

One thing to remember is that the p-value is a statistic---it takes your data and transforms it in a particular way. As a result, the p-value is a random variable. It has a mean and a variance. It is difficult to calculate the distribution of the p-value generally, but it is easy to calculate it under the null hypothesis: if the null hypothesis is true, the p-value has a uniform distribution from 0 to 1.

Since the p-value is random, it's possible that the two p-values that you are comparing are not statistically different. (Again, calculating p-values for differences in p-values is hard, typically requiring stronger assumptions than the problem gives. That two p-values are statistically the same, then, is hard to test.)

Andrew Gelman likes to say that "the difference between statistically significant and not is not statistically significant." In other words, 0.05 is not a magical cutoff and being below this value isn't necessarily different than being a little above due to the randomness of the p-value itself.

Another issue is that some procedures are more efficient than others. Typically, efficiency is gained by making additional assumptions about the data and using those assumptions to devise tweaks in the estimation procedure. It's possible that the two studies have the same estimate of the effect, but have different standard errors, leading to different p-values simply because one study makes additional assumptions that gets it lower standard errors.

Of real interest to you should be whether the two studies estimate the same effect. If the estimates are similar, most people would choose the more efficient/significant one unless they believe that the assumptions needed for the efficiency boost are not valid.

If the estimates of the effect itself are not similar, then you need to look at the two estimation procedures and determine which is more compelling---this isn't something that I can comment on without knowing the specifics.

-
+1, nice answer –  mpiktas Oct 3 '11 at 12:45

Quite possibly BOTH studies are correct. Also possible that both are wrong. Or that either is correct.

What question did the two studies try to answer? What statistical techniques did they use? What were the parameter estimates for each? Did they use the same data? What data did each use?

Right now, your question is something like: One person says Scotch is better than beer. Another says it isn't. Who is right? They use different taste buds.

-
I have no idea why people are voting this answer down. –  EpiGrad Oct 3 '11 at 7:21

Assuming that both studies are testing the same null hypothesis, your question exposes an important problem with Neyman-Pearson type hypothesis testing. If we don't take the first result as final then we are not 'playing by the rules' and we don't control the rate of false positive results. That is why Fisher repeatedly complained that the N-P method was good for industrial quality control and acceptance procedures but all but useless for scientific enquiry.

If you really want a composite result from the two studies then you need to use the exact P values rather than the N-P binary division between significant and not significant. Google 'combining p values' to get a start (I'm on leave playing with a iPad so I don't have access to my resources, so that's the best I can do for you at the moment.)

-

There isn't really a credible "Wrong" and "Right" study based on p-values. You can run a shoddy study and get a really low p-value (indeed, its easy to do), or you can run run a study that is essentially a scientific masterpiece, and get a non-significant result. Or indeed you can run the same study multiple times, and have a spread of significant and non-significant results.