2
$\begingroup$

I'm doing a logistic regression in R (using glmer and family='binomial').

I'm measuring how performance in a task differs between 2 factors (2 levels each). The IVs are categorical. (It might be helpful to know that the design is split-plot, with one IV repeated within subjects and subjects nested within the other IV).

I previously did the analysis as a regular GLM by converting the DV to percent correct (i.e. number of items correct / number of items total) for each condition. However, this method does not allow for item to be used as a random effect. It also assumes a continuous distribution for the DV and a distribution that goes above 1.0 and below 0.0, both of which are violated.

Therefore, I chose to use logistic regression and code each item (per participant) as a success or failure (DV is 1 or 0). (Subject and Item are random factors in the analysis).

Interestingly, the logistic regression completely eliminates the interaction effect of the 2 IVs, which was significant in the linear regression. (This is even the case if I make the analyses more comparable by not adding item as a random effect.)

One reason why I think this is happening: One of the 4 categories has a very high proportion of participants with perfect accuracy (i.e. 10 of 10 items correct). It seems that this is disproportionately increasing the Beta estimate for that category.

Looking at a graph of the logit function, the mapping between probability and log odds ratio blows up around p=1.0 or 0.0, with very small changes in p producing very large changes in the log odds ratio. (The R analysis uses a binomial distribution, but I would think this still to be related.)

In cases like these, with a categorical IV and perfect accuracy for one of the categories, is logistic regression still valid? My intuition is that this might make the estimates less robust. If so, what are alternatives to logistic regression that I can perform which will still counter the problems associated with regular GLM?

$\endgroup$
1
$\begingroup$

All 1s or 0s for some categorical independent variable (or combination of them) in standard logistic regression would be a case of “complete separation”. I.e. the maximum likelihood estimate of the log-odds ratio is not finite and any output from a software is suspect. The random effects in the model might make some difference to what should and does happen, but I don’t think so.

Standard strategies when this happens are Firth’s penalized LR, exact methods - but these do not allow random effects -, or Bayesian methods with (weakly-)informative priors. There may be other reasonable approaches that I am not aware off, but the last one one would be my personal choice. In terms of implementation this may require specifying the model in your Bayesian language of choice (Stan, JAGS, PROC MCMC, BUGS etc., which have interfaces to many platforms/programs such as R, SAS, Python etc.), unless the particular model is already implemented somewhere.

$\endgroup$
1
$\begingroup$

As @Bjorn as pointed out this is called separation.

One other way of approaching this is to ask what it means in the context of your scientific question. Why do you get a perfect prediction here? Is this (a) just the result of your not having very much data in that part of data space or (b) does it reflect something interesting about the world, or (c) is it just an uninteresting feature of the way the variables relate which is an artefact? If it is (a) then @Bjorn's suggestions may help you but if (b) or (c) you may need to find some way of describing what happened and perhaps even take the extreme step of excluding that level of that factor from your model. This is an interesting situation and I do not think there is a single correct answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.