1
$\begingroup$

When I fit a binary predictor in the logistic regression model, I got a high odds ratio and a very wide 95% CI (e.g. OR=11, 95% CI [1.5-229.5]; p = 0.041). My binary predictor is smoking status and my outcome is having the disease or not.

The distribution is as follows: smoking+disease: n= 7; smoking+non-disease: n= 1; non-smoking+disease: n=7; non-smoking+non-disease: n = 11

I have never seen this wide 95% CI before and believe it is not correct. What should I do next to fix this issue?

$\endgroup$
1
  • $\begingroup$ The confidence intervals being wide is what makes them correct (or at least approximately correct; see below). Like relative risks, odds ratios for rare outcomes are highly unstable. $\endgroup$ Nov 22, 2021 at 13:50

2 Answers 2

1
$\begingroup$

Without seeing your actual data and especially what tool you are using, it is very hard to give a conclusive answer. Note that calculating confidence intervals for odds ratios is not straightforward and usually involves some large-sample approximations. These, in turn, are likely not warranted with your small dataset, so your tool may be doing something else, perhaps some simulation/resampling approach.

If we feed your data into R and use the epiDisplay package,

dataset <- data.frame(
    disease=c(rep(TRUE,7),rep(FALSE,1),rep(TRUE,7),rep(FALSE,11)),
    smoking=c(rep(TRUE,7+1),rep(FALSE,7+11)))
model <- glm(disease~smoking,data=dataset,family="binomial")
library(epiDisplay)
logistic.display(model)

we get this:

Logistic regression predicting disease 
 
        OR(95%CI)        P(Wald's test) P(LR-test)
smoking 11 (1.1,109.67)  0.041          0.016

We note that the point estimate of the OR is the same as yours, $11$, which is unsurprising, since the point estimate relies on the parameter estimate only, and there is little leeway in estimating this. The uncertainty comes in in estimating the confidence interval, and we note that our output above gives you a different CI than the one you post in your question.

However, our CI is still rather wide, and you might have asked the exact same question here if you had seen a CI of $[1.1, 109.67]$ in your tool's output. And the problem is the same, regardless of your tool: it's that you have very little data indeed. Remember that the OR measures the multiplicative increase in the odds of having the disease between a smoker and a nonsmoker. Thus, it relies on all four entries in your cross-table. And the precision by which we can estimate it, as a rule of thumb, depends on the smallest entry. Note that there is only a single smoking and non-diseased participant. If there had been just a single more such person, your estimated OR would have changed drastically, halving to $5.5$ (and of course the CI would have shrunk dramatically, too)! Thus, there is just a huge uncertainty in estimating the underlying relationships from small datasets, and the CI correctly reflects this in being very wide, no matter what approximation we use.

(Incidentally, that is also why we should take your statistically significant result with a large grain of salt.)

The only remedy is to collect more data. Ideally lots more data. I understand that may not be feasible for you, but matters being the way they are, you can simply not conclude a lot from the data you have - and that the CI correctly shows this is definitely a feature, not a bug.

$\endgroup$
4
  • $\begingroup$ Hi Stephan - thank you for the detailed response. I saw some papers including upper bound 95% CI as high as ~30. Do you know what's the recommended range of the upper bound so we can include it in the results? $\endgroup$
    – R Beginner
    Nov 22, 2021 at 18:12
  • $\begingroup$ I don't think there is a generally accepted range. The smaller your sample size, the larger your CIs will be. If you plan on submitting this analysis for publication, I would recommend that you have a very good reason for going with this small sample size, and/or explicitly label your results as tentative. $\endgroup$ Nov 22, 2021 at 18:29
  • $\begingroup$ Hi Stephan - thank you for the suggestion. Is it acceptable to report the 95% CI as [1.5-Inf] where Inf stands for infinity? $\endgroup$
    – R Beginner
    Nov 24, 2021 at 3:59
  • $\begingroup$ That would be very uncommon, although a case could conceivably be made: such an improper interval would be similar to a one-sided test. However, you should anticipate some very probing questions, both from the non-statisticians ("What is this?") and from the statisticians ("How do you calculate that?"). Yes, these can be answered, but why would you want to go to the trouble? Easiest to just report the CI your tool yields - the precise upper limit is not really that important. $\endgroup$ Nov 24, 2021 at 6:33
0
$\begingroup$

Your sample size is very small, I would not be surprised to see such a wide CI. E.g. if you have 10x the size with the same proportions of smoking/disease and change your code accordingly, your ORs will be 11 (95%CI 5.32-22.76). You should probably do some power calculations for your study to check if it is enough to make any conclusions.

$\endgroup$
8
  • $\begingroup$ Hi Diana - thank you for the feedback. Do you think if I can use Fisher's exact here to report the 95% CI (i.e. I will report OR of 11 (from logistic regression) but with the 95% CI from Fisher's exact)? Is this a valid approach? $\endgroup$
    – R Beginner
    Nov 22, 2021 at 18:15
  • $\begingroup$ So-called "post-hoc power calculation" will not be helpful (Hoenig & Heisey, 2001). What would be helpful would be a power and sample size calculation prior to re-running the study with a much larger sample size - essentially using the present dataset as a pilot study that can be used to determine likely effect sizes in a power calculation for a "real" study. $\endgroup$ Nov 22, 2021 at 18:32
  • $\begingroup$ Got it! Thank you Stephan :) $\endgroup$
    – R Beginner
    Nov 22, 2021 at 18:49
  • $\begingroup$ So, it is not valid to report the mixed results (using OR from logistic + 95% CI from Fisher's exact)? $\endgroup$
    – R Beginner
    Nov 22, 2021 at 18:50
  • $\begingroup$ @RBeginner I used the code by Stephan above, which uses logistic regression. CI is estimated from normal approximation, so for a small sample like yours Fisher test may be a better option indeed. I see somewhat different results in R though: dat <- data.frame("smoke_yes" = c(7, 1),"smoke_no" = c(7, 11),row.names = c("disease", "non-disease")); fisher.test(dat) gives OR = 10.03, 95% CI 0.96-537.80. And you should report OR and CI from the same model $\endgroup$
    – DianaS
    Nov 23, 2021 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.