I hope I am able to word my question clearly. Suppose I have a model below:

glmer(Y~X + (1|subject), family="binomial", data=dat)

The intercept is the log odds of success for the reference level (level 1 of X), and the slopes indicate the log odds ratio of the other levels to the reference level.

            Estimate Std. Error z value Pr(>|z|)    
(Intercept)   2.1745     0.3517   6.183  6.3e-10 ***
         X2  -0.5559     0.3276  -1.697   0.0897 .  
         X3   0.2309     0.3634   0.635   0.5252    
         X4  -4.8587     0.4155 -11.693  < 2e-16 ***
         X5  -2.8946     0.3200  -9.045  < 2e-16 ***
         X6  -3.6111     0.3387 -10.663  < 2e-16 ***
         Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

This is all fine. However, what kinds of statistical tests can I use if in addition to this, I would also like to check whether each level (including the reference level)'s odds of success are significantly higher than 0? How would I be able to do this?


You fit a binomial probability model with - by default - a logit link function (see ?binomial). The problem is that the logistic function (the inverse of the logit) never reaches zero. Therefore, all model fits will yield nonzero odds for success. In particular, all classical confidence intervals for success odds will not include zero. Therefore, in classical Null Hypothesis Signifiance Testing (NHST), your odds of success will always be (statistically) significantly different from zero. (For that matter, the same applies to one.)

Your best bet will probably be to abandon statistical significance and go with practical significance. Determine beforehand what "relevant" odds of success are, then look which predictor levels yield a fit that exceeds this level.

Which leaves you with the problem that prediction for mer objects is not a straightforward matter. This question may be helpful here.

Good luck!

  • $\begingroup$ Thank you for the reply! Regarding "the logit never reaches zero", could you please clarify that? Wouldn't the logit of 1 (namely when the odds of success are 1 (e.g., 0.5/(1-0.5)) be 0? $\endgroup$
    – Alex
    Feb 12 '13 at 14:50
  • $\begingroup$ Sorry, I corrected the error in my answer - thanks for keeping me honest! Of course the logit can reach zero. The problem is that its inverse, the logistic function, from which we calculate fitted values, will not reach zero. Hope it is clearer now? $\endgroup$ Feb 12 '13 at 14:58

If you really want to test the null hypothesis that the true odds for success are exactly zero for each individual level $j$ of your factor $X$, then the statistical test is very simple.

The null-hypothesis odds $p_{0j}/(1-p_{0j})$ for success in group $j$ are zero iff $p_{0j}=0$. Under this null hypothesis, the probability of observing at least $1$ success in group $j$ is $0$. Therefore, if you empirically do observe a success in group $j$, you can reject the null hypothesis for any level of $\alpha > 0$.

Since this decision rule allows us to set $\alpha$ arbitrarily close to $0$, we do not have to worry about $\alpha$-inflation when doing multiple tests. So: for each group $j$ with at least one success, you can reject the null hypothesis of $p_{0j} = 0$.

If this test seems odd, or somewhat non-informative, it could indicate that you are actually not really interested in whether the true odds are exactly $0$, but just very small. In that case, I seond Stephan's remarks on statistical significance vs. practical relevance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.