-1
$\begingroup$

Statistical significance indicates that the difference between 2 variables is real and not due to chance. It doesn’t indicate that the difference is large. But isn’t saying that a difference is real is also implying that the difference is large enough to be real?

$\endgroup$
3
  • $\begingroup$ Your first sentence is badly mistaken. No statistical method can test for 'realness'. Perhaps see a few of the questions on this site that have already been answered. Maybe start here: stats.stackexchange.com/questions/17910/… $\endgroup$ Aug 12, 2016 at 6:38
  • $\begingroup$ Statistical significance, $p$-value, and confidence intervals have very exact and thus limited meaning. This may help you: dx.doi.org/10.1080/00031305.2016.1154108 $\endgroup$ Aug 12, 2016 at 7:35
  • $\begingroup$ The relevant xkcd are xkcd.com/882 & xkcd.com/1478 There are lots of proposed examples about some test that 'beg the question' by using words that pre-imply certain expectations and then hypothesis a different outcome. The big question is why "1 in 20" is (or isn't) suitable as a way of deciding if the results follow expectation. If you really have a fair coin, then 4 heads (or tails) in a row is easily possible, but 10 in a row in unlikely unless the coin was not fair (unless you have 200 students in the lecture hall); go figure... it's interesting;-) $\endgroup$ Aug 12, 2016 at 10:38

2 Answers 2

3
$\begingroup$

I suppose the gist of this has already been answered in the comments, and I understand why the commentators have (perhaps impatiently) referred to previous answers. But as a newbie I have a little more patience, so let me elaborate a little.

A. Statistical significance
1. A significant result does not indicate that the result is real. It simply indicates that if the null hypothesis were true (if there were no effect in the population, or to put it another way, if the two groups were drawn from the same population in terms of the thing you were measuring), you would have seen a result this impressive 5% of the time or less.
2.In more detail, suppose the null hypothesis were true. You take samples (let's say 20 people each) from the same population and measure them on some attribute, e.g. fine motor skills. Because everyone is different, the means in the two samples would not be the same as each other (or as the mean in the population). This is called sampling error. (I know that other people will read this, so I will acknowledge that sampling error has a slightly more exact definition which I don't think matters here.)
3.Statistical tests (certainly for the difference between two means) use various clever maths to work out what would happen if we repeated the study time and time again, and hence what would be an unusual size of the difference between the means.
4."Unusual" is defined in terms of how often a result that impressive (in the case of a difference in means, that big or larger) would happen if the null hypothesis were true. This is called the "alpha level" and is usually set at 5%, which is what I have presumed in the above explanation.
5. Even then, 5% is not that impressive. If you did 20 statistical tests where there was no effect and with a 5% alpha level, you would expect 1 of them to be significant. (That doesn't mean that exactly one will be significant; the chance of none of them being significant is a slightly more complicated calculation which I don't think we need to bother with.)
6. Even then, this is the percentage chance of getting the result if the null hypothesis were true. Contrary to what you might think, that still does not make it the chance that the null hypothesis is untrue.
7. I heartily recommend Cohen's classic 1994 paper "The earth is round (p<.05)" to take the issue further. I think Cohen was even more fed up than some of the people commenting on this question.

B. Is the result "large" enough?
1. The chance of getting a significant result, if the effect is real, is called "power". This depends on various things, including the size of the effect and the sample size.
2. If you have a large sample size, you could easily get a significant result even if the effect is very small. For example if I test a new headache pill against aspirin, and people taking the new headache pill get 1% fewer headaches, I might get a quite correctly statistically significant result. But this is not a difference that patients are likely to notice. It would be statistically significant, but of clinical or practical significance.

$\endgroup$
1
3
$\begingroup$

Statistical significance doesn't mean what you think it does.

Firstly, it is not any guarantee that it's "not due to chance". If $H_0$ is true the probability of observing a statistic at least as extreme as the one you had is low, but clearly possible (how low depends on your choice of significance level).

When a test statistic is unusual (in the sense of being in a part of the distribution under the null - generally a tail - that's most highly consistent with the alternative) we're left with two competing explanations. Either

  • the null is true and something quite rare happened

  • the null isn't true (and so there's no need to invoke a rare fluke as an explanation)

If we reject $H_0$ what we're really saying is that the difference was large enough that we could reasonably conclude it wasn't exactly 0.

But isn’t saying that a difference is real is also implying that the difference is large enough to be real?

This is the fallacy of equivocation -- you used the word "real" in two different senses. Let's avoid the word "real" and see what you're saying:

But isn’t saying that we can detect that the difference is not zero also implying that the difference is large enough to be important?

The answer to that question is hopefully now clear - no.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.