For getting to know statistics better, I replicated an existing study (same questions) and did the survey (n=66). I calculated correlations (Pearson bivariate, 2 sided) and found statistically significant results. Unluckily, some of my results are insignificant where the first study (which I replicated) found significant results.

What can I say about that (besides that I have to reject the hypothesis and I would need a larger sample size to get significant results)? Is the first study wrong?

  • $\begingroup$ No reason to feel stupid or confused. Your data give different results; that's the nub of it. There is no reason, from what you say, to regard either study as wrong. Depending on your field, it may be that the first study was only regarded as publishable because it got (more) significant results. $\endgroup$
    – Nick Cox
    Aug 13 '13 at 9:11
  • $\begingroup$ If you calculate a 95% confidence interval of the correlation coefficients for your and their study, you'll likely still observe overlap, and it will give you a feel for which range of values either study is compatible with. $\endgroup$
    – jona
    Aug 13 '13 at 10:03

The first study isn't wrong, nor is your study - one can get "significant" and "nonsignificant" results from the same study. I find the best way to imagine this is to think about the correlation you're researching as having a "True" value and distribution out in the universe, one that - not being omniscient - you cannot know. Each time you conduct a study, you make a single random draw from this distribution, each time getting a slightly better picture of the "True" distribution.

Imagine the True value of your correlation is normally distributed, and centered around 0. If you ran infinitely many studies over and over again, 2.5% of the studies you run will give you statistically significant results that say it's negatively correlated, and 2.5% of the studies you run will give you a statistically significant result that say it's positively correlated. The rest will report zero, negative or positive correlations with p-values ranging from 0.051 all the way up to 1.00.

None of those studies are wrong. They just came from a different part of the distribution - they're different draws, and each new study represents a slightly better picture. That's why repeatability is so important. Some of your evidence says there's a correlation - some of your evidence says there isn't. All you can really say definitively right now is that the evidence doesn't support that there is a correlation, and that more evidence is needed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.