I want to run a logistic regression with a binary outcome (correct vs incorrect) and three predictors: condition (2 levels: A and B), and time (2 levels: before and after), and their interaction. I've run into a problem, however, in that the model gives meaningless parameter estimates with huge SEs. I'm pretty sure the reason is that for one level of the condition predictor the outcome is always correct. Alternatively, it could be that there is some missing data (see below) but my understanding is that this shouldn't be a problem for logistic regression.

The counts are:

  • Cond A T1: 28 of 28 correct
  • Cond B T1: 23 of 26 correct
  • Cond A T2: 30 of 30 correct
  • Cond B T2: 24 of 30 correct

Is there a way I can run this analysis? The reason being that if I just use chi-square to compare the counts then it looks as though at T2 there is a significant difference between Cond A and Cond B, but this doesn't account for the fact that Cond B was already doing worse at T1. I'd also like to test for a main effect of condition.

EDIT: I realize I didn't model the within-subj variance, which I can do with this code by adding it as a random effect in a mixed model:

glmer(df$outcome ~ df$condition*df$time + (1|d$ID),       family=binomial(logit))

However I get an error message that the "model is nearly unidentifiable: large eignenvalue ratio". Additionally, the estimates are identical whether I use this model or the one where I didn't model within-subject error as a random effect, which made me wonder if I was doing something wrong. There was, however, some variance associated with this term so maybe it's fine (aside from the error message)?


As answered above, you seem to be handling within-subject correlation by assumption it's zero.

But to the original question there is no reason to abandon the model or the parameter estimates in the case of complete separation. The estimations of $\beta$ are still valid. As stated above they will result in probability estimates of 0 or 1. Standard errors are not valid. So you have the Hauck-Donner effect. You can still get completely valid hypothesis tests using likelihood ratio $\chi^2$ tests, ignoring the standard errors.

| cite | improve this answer | |
  • $\begingroup$ Thanks!Please see my edit above re: within-subj correlation and let me know if that makes sense. Re: separation, so if I use a nested model comparison approach I can get the correct likelihood ratio tests instead of relying on the z scores for the param estimates? I'm questioning whether the Bs are correct because I should be able to exponentiate my intercept to get the odds relative to chance, but the value is ridiculously large vs. when I run a model estimating just the intercept and compare it to a model not estimating it. $\endgroup$ – PanPsych May 22 '16 at 15:38
  • $\begingroup$ Correct; ignore z-statistics and standard errors. Likelihood ratio tests are OK. Anti-logging intercept is only meaningful if you want to get the odds that $Y=1$ vs. $Y=0$ when all $X=0$ (e.g., if only categorical $X$s are in the model and one sets all the categories to reference categories). "Relative to chance" isn't perfectly well defined. $\endgroup$ – Frank Harrell May 22 '16 at 16:08
  • $\begingroup$ Okay, good to know! To clarify, the estimates for the predictors are interpretable (can be exponentiated to give odds ratios)? $\endgroup$ – PanPsych May 22 '16 at 16:12
  • $\begingroup$ For example, I just ran the model comparison testing the main effect of condition and while the likelihood ratio tests seem to make sense, the estimate for condition is -38.99, which doesn't convert to an interpretable odds ratio (1.15e-17). Same for the estimate of time in a separate model comparison (logit = 3.64) but the likelihood ratio test comparing models with and without that predictor is highly non-significant. $\endgroup$ – PanPsych May 22 '16 at 16:24
  • 1
    $\begingroup$ Agatin, make sure you handle time correctly. But to your question the odds ratio estimate is $\infty$ which is legitimate if a cell probability is 0 or 1. $\endgroup$ – Frank Harrell May 22 '16 at 16:54

Logistic regression seems wrong as it would assume independence between the observations at time t1 and t2 (I assume these are the same subjects). Either t1 could be a factor in the model for t2 (or vice versa) or you would want a generalised linear mixed model (GLMM) or could use GEEs. Also, you probably cannot ignore the missing data of people with a response at t2 that do not have a response at t1 except under some extremely strong assumptions (MCAR). A GLMM would implicitly impute the missing data under a specific MAR assumption, which at least allows the response at one time to influence the missingness at the other time.

Regarding the separation, one thing that would help is a Bayesian approach with at least weakly-informative priors.

| cite | improve this answer | |
  • $\begingroup$ Thanks. Please see my edit above. Things don't change when i use the mixed model. $\endgroup$ – PanPsych May 22 '16 at 15:41

The other answers are helpful from a general perspective, but there is an additional consideration for this data set: it's too small.

Note that you do not really have 114 informative data points from the perspective of logistic regression. The effective data size in logistic regression is the smaller number of the 2 outcome classes. That's only 9 in this data set (the "incorrect" cases). Based on the rule of thumb of 10-20 of the least-frequent case per predictor variable evaluated, that's barely enough to support analysis of the main A versus B effect. Treating the 30 individuals as a random effect in a mixed model adds at least 1 more predictor to the model (if only differences among intercepts are of interest) and adds 2 predictors if you also care about differences in slope (as you seem to, given your initial attempt to look at the treatment/time interaction). So you really need about 3 times as much data to accomplish what you want.

| cite | improve this answer | |
  • $\begingroup$ Thanks. So to clarify, this analysis a manipulation check to see if the participants remembered key information about the task (the outcome variable) and to check whether this varied by condition or the timing of the question. My sample size for my primary analysis is fine, but I wanted to run this as well. Is your comment about power to detect the interaction effect? $\endgroup$ – PanPsych May 22 '16 at 15:49
  • $\begingroup$ You may in principle run into problems with power or with overfitting when there are few of the least-frequent class in a logistic regression. I suppose one can think of the Hauck-Donner effect as an overfitting that provides problems with both: "perfect" fitting that may be unrealistic, and enormous standard errors of coefficients. $\endgroup$ – EdM May 22 '16 at 23:10
  • $\begingroup$ (+1) But though the H-D effect quite commonly arises from fitting too many parameters for the amount of data you have, it can just as well arise when over-fitting isn't an issue at all. And it can't be said often enough that the enormous standard errors are solely the result of a poorly chosen approximation - profile likelihood CIs are much better. $\endgroup$ – Scortchi - Reinstate Monica Jun 6 '16 at 20:03

Yet another approach would be to use Firth's bias-corrected logistic model. It is outlined in

  author = {Firth, D},
  year = 1993,
  title = {Bias reduction of maximum likelihood estimates},
  journal = {Biometrika},
  volume = 80,
  pages = {27--38},
  keywords = {glm}

and is available in R and Stata at least.

I am not sure that the sample size you have is a issue at this point, you would not design a study to have this size knowing the prevalence of your outcome but now you have the data you have to do the best you can with it.

| cite | improve this answer | |
  • $\begingroup$ Can you explain why this would help? Judging from at least one other response here, I might still be able to use (mixed) logistic regression. I don't think my sample size is a problem. My main response measure is different and this is essentially a manipulation check to see if the participants remembered key information about the task and to check whether this varied by condition or the timing of the question. So I'd expect the prevelance of the correct response to be high no matter what the sample size. $\endgroup$ – PanPsych May 22 '16 at 15:46
  • $\begingroup$ I now see that you in fact have a mixed model so I do not think it will help, sorry. $\endgroup$ – mdewey May 22 '16 at 15:49

Your diagnosis is correct. Since you are using a model with two main effects, one interaction, and (I presume) one intercept, you have as many parameters as conditions. Your model will fit itself exactly to the empirical results under each condition: it will predict a probability of 28 / 28 = 1 for Cond A T1, et cetera. For a logistic regression to predict a probability of 1 or 0, some combination of the coefficients must go to the logit of 1 or 0, which is $\pm \infty$.

It doesn't seem profitable to analyze these data on a logit scale. I'd suggest you look instead at the the risk difference between the groups. You can test first for a difference between condition A and condition B using a simple z-test like this. You can also test different times within condition B with same method.

One warning: the normal approximation may not work great given your low counts.

In my opinion, there is not too much point in looking for a main effect of time. On a risk-difference scale, the model with intercept, condition main effect, and a condition B : T2 indicator can explain as much of the observed variation as anyone possibly could with those covariates.

| cite | improve this answer | |
  • $\begingroup$ Thanks. I'm not sure I follow but let me clarify that this same problem with the model results being uninterpretable occurs when only the intercept and condition are in the model. And I have 114 data points. But when I have a model with the intercept and time in it, the model works fine. What I'm not clear on is why exactly it doesn't work when condition is a predictor. Condition A always yields outcome 'correct', but I'm not sure why that is a problem? $\endgroup$ – PanPsych May 22 '16 at 3:12
  • $\begingroup$ Try calculating the observed odds: 30/30 / (1-30/30). $\endgroup$ – Björn May 22 '16 at 5:28
  • $\begingroup$ The same principle applies. The model will fit condition A and condition B separately, since you have both an intercept and a condition main effect: $$p_A = expit(\beta_0)$$ $$p_B = expit(\beta_0 + \beta_{cond})$$ For a logistic regression model to approach a predicted probability of 1, which it will for condition A, the combination $\beta_0 + \beta_{cond}$ must go to infinity because that's the only "value" whose expit is 1. $\endgroup$ – eric_kernfeld May 22 '16 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.