2
$\begingroup$

Just when I thought I was beginning to understand the concept of statistical significance, I learn about odds ratios and everything seems to get thrown out the window.

My understanding of p < 0.05 was as follows: Given that the null hypothesis is actually true, p < 0.05 indicates a less than 5% chance of getting this result with this sample size. This somehow ties in with having a confidence interval of 95%, I'm still working on getting the intuition there.

In writing this question I actually am beginning to see the link between what I just learned with odds ratios and the whole p < 0.05 thing. I'm still not there though. Hopefully someone can spot the gap in my intuition and help fill it in. I hope this question isn't too vague - thanks in advance for any help you can provide.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

They can be related in some cases (see comments). The p-value is formally defined as the probability under the null hypothesis of observing the same or a more extreme result than that in your sample. 0.05 represents a rule of thumb cutoff for this value to be considered significant.

The confidence interval is a plausible range of numbers for the true population value of the parameter of interest. Formally, if one collected a sample and calculated a 95% confidence interval, then repeated collection and calculation a further 99 times, 95 of those 100 intervals are expected to contain the true value of the population parameter.

The link between p-values and confidence intervals is this: if you test, for example, the null hypothesis that a certain parameter, say the population mean, is different from 0, and reject this null hypothesis because you calculate from your sample a p-value < 0.05, the 95% confidence interval calculated from this sample will not overlap with 0. Similarly, if we fail to reject the null hypothesis (i.e., calculate a p-value > 0.05), then the 95% confidence interval will overlap with 0.

For odds ratios, it's a little different, because testing if the odds ratio is statistically significantly different from 1 is the same as testing if the two different groups have different odds. So usually the null hypothesis tested is that the odds ratio is equal to 1, and we check to see if the 95% confidence interval overlaps with 1. If you reject the null hypothesis (at a significance level of 0.05), then the 95% CI won't overlap with 1.

ADDENDUM: this relationship holds because for a given significance level $\alpha$, the corresponding confidence level is 100%(1 - $\alpha$), so for $\alpha$ = 0.05, 100%(1 - 0.05) = 95%

EXTRA ADDENDUM: Alexis correctly pointed out that this only holds for certain tests and CIs, and isn't a general rule.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ I think you should stress that coverage of an estimate by CIs does not in general correspond to $p > \alpha$, that this is true only for specific tests/CIs. In fact there's a "visual hypothesis testing" literature on precisely this issue. See for example, Afshartous, D. and Preston, R. (2010). Confidence intervals for dependent data: Equating non-overlap with statistical significance. Computational Statistics & Data Analysis, 54(10):2296–2305, and Cumming, G. (2009). Inference by eye: reading the overlap of independent confidence intervals. Statistics In Medicine, 28(2):205–220. $\endgroup$
    – Alexis
    May 17, 2018 at 16:15
  • $\begingroup$ @Alexis just to clarify, for my own understanding, is this in general for the parameters, or for confidence intervals on the parameters between two populations? So for example if the CIs for the mean of two independent populations overlapped but they were statistically significantly different at $\alpha$ = 0.05, would the CI on the difference in means contain zero? $\endgroup$
    – NatWH
    May 17, 2018 at 16:23
  • 1
    $\begingroup$ It is going to depend on the specific test and specific confidence intervals. For example a proportion is the mean of a 0/1 variable, but the relationships between a z test of proportion difference and the exact binomial CI, Wald CI, or the Agresti-Caffo CI is different than the relationship between a t test of mean difference and a Wald CI. $\endgroup$
    – Alexis
    May 17, 2018 at 16:31
  • $\begingroup$ @Alexis I have removed that sentence so as not to contradict. $\endgroup$
    – NatWH
    May 17, 2018 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.