3
$\begingroup$

I think have a misconception about statistical significance.

For example, I have two treatments A and B. Mean effect of A is -105 with 95%CI:(-168,-42) and for B -88 with 95%CI: (-146, -29). Which of the results is more statistically significant then?

When can we conclude in general that result is more statistically significant based on p-values or confidence intervals?

However I've read here that

A result does not become "more" statistically significant if the p value is "a lot smaller" than the alpha value, as opposed to being simply "slightly smaller".

So is it correct to compare statistical significance in such a way?

I'm very confused, so I appreciate any hints.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ In some contexts, with some interpretations of "statistical significance," this question makes no sense at all (as hinted by your quotation) and in others it is fine but cannot be answered with the information given. What, then, do you mean by "statistically significant"? $\endgroup$
    – whuber ♦
    Commented Jan 11, 2014 at 15:12
  • 1
    $\begingroup$ As I understand statistical significance in quotation means a probability to reject true H0, which is fixed e.g. 5% and I don't understand how it relates the significance of effects. In treatment example I have several versions what 'more significant' means: 1) the effect is less likely to be zero 2) the magnitude of the effect is bigger 3) the width of confidence interval is smaller. I should have started from the question that I don't understand what is significance here. So I would be very grateful if you can tell me that. Thank you very much! $\endgroup$
    – user2575760
    Commented Jan 11, 2014 at 15:29
  • 1
    $\begingroup$ You'd have a much better question incorporating this list of potential meanings into it. Perhaps keep that in mind for the future. According to the website you're reading, none of your proposed interpretations of "more significant" is correct. It's best to just never use a magnitude qualifier in front of significance because, even when it's justified, there are much clearer ways to say things. $\endgroup$
    – John
    Commented Jan 11, 2014 at 16:03

2 Answers 2

Reset to default
5
$\begingroup$

Of course you're confused. The web page you're reading is confusing. The opening sentence is incorrect and the opening paragraph is riddled with problems primarily focusing on a magnitude of significance. In fact, the only paragraph on the page that seems satisfactory to me is the one from which your quote is taken.

Nevertheless, there is a consistency. If significance were to have an amount it's not the p-value, it's the alpha value. And, the meaning of alpha is explained there well. Therefore, according to the site, one can infer that the amount of significance can only change due to lowering the alpha value before the study is run and the discovered p-values or CI's make absolutely no difference. The latter part of that is correct. But, generally it's advised that there just really isn't such a thing as more or less significant. I don't know anyone personally who would endorse that website's interpretation of the term "more significant".

It appears, from your question, that you want to say one of your tests is more significant than the other. You cannot do that, even following the referenced website, because you calculated 95% CI's for both and therefore used the same alpha values (alpha is used to determine the % of the CI). You can't change alpha afterwards. Furthermore, it's quite unlikely that you even want to say "more significant" for treatment A. You would probably like to say that it had a larger effect. In that case, what you would do would be to compare the effects of treatment A and treatment B directly.

The test comparing treatment A and B would fail to show they were different. That's evident in the large overlap in your confidence intervals. I'm suspecting they're fairly wide CI's as well. I say "suspect" given that the width is substantially greater than the magnitude relative to 0. I'd need to know much more about the data to make a confident qualitative statement that they are wide because it depends on several factors. However, given the apparent width, the best thing you can say is that the CI's of A and B largely overlap and it cannot be determined from these small samples whether the observed differences are simply due to variable samples because those CI's are also rather wide.

Improve this answer
$\endgroup$
6
  • $\begingroup$ The problem is that I don't want to compare treatments directly. I had this question on exam: is the effect of A is more statistically significant then effect of B? So I don't even know how to interpret it. If you know, please help me. Thanks a lot! $\endgroup$
    – user2575760
    Commented Jan 11, 2014 at 16:03
  • $\begingroup$ Have your instructor assassinated (not a recommendation for a genuine assassination). We'd really need to know what your instructor thinks that "more significant" is in order to answer this. If they're using rational wiki then it means you changed alpha between tests and the new alpha was passed. However, I'm guessing (hoping) it's just a trick question because your instructor has already told you not to think in terms of "more significant". $\endgroup$
    – John
    Commented Jan 11, 2014 at 16:04
  • $\begingroup$ Unfortunately, I didn't ask my instructor this question, so I don't know if I really shouldn't think in terms of 'more significant'. $\endgroup$
    – user2575760
    Commented Jan 11, 2014 at 16:30
  • $\begingroup$ Then ask your instructor. And, in the future, when asking questions here give more information. You'll get better answers. $\endgroup$
    – John
    Commented Jan 11, 2014 at 16:46
  • 1
    $\begingroup$ In that case the argument comes partly because the 5-6 is narrower and partly because the lower bound is a greater distance in proportion to the width from 0. Consider that a CI is usually about 4 standard errors wide then the narrow one is 20 SE's from 0 while the broad one is not even 1. It's not a universally accepted argument and may not have been what your instructor was looking for. $\endgroup$
    – John
    Commented Jan 11, 2014 at 17:14
6
$\begingroup$

It's simply that "significance" is used in hypothesis testing to refer to a p-values' being smaller than a pre-determined value ($\alpha$ here). So you don't say results are "more" or "less" significant just as you don't say Jack passed an exam "more" or "less" than Jill.

That doesn't stop you considering smaller p-values as indicating greater discrepancy with the null hypothesis; they're a monotonic function of the test statistic you chose to measure just that.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.