Are residuals important in logistic regression when all variables are factors?

Question

I have a conceptual question about residuals in logistic regression models.

I understand that for linear regression, the residuals represent the differences between the observed and fitted values. A linear regression model can be evaluated in terms of how much the fitted values deviate from the observed values (the deviance), and that there are various statistics use the deviance to evaluate goodness of fit, like AIC one of the many R2 statistics.

I can see how this is the case in logistic regression wherein at at least one variable (either response or explanatory) is continuous.

What is less clear to me, is the role of residuals where all variables are factors.

So say I have a model that tries to predict the choice of outcome A over outcome B, and the explanatory variables are all binary or otherwise have factorial levels "yes, no" or "option 1, option 2, option 3".

In such a case, what is it that the residuals represent? There's not really a lot of "distance" between values right? Seems to a bit more abstract to me.

Most of the resources I've read about logistic regression involve examples with at least one continuous variable, very few deal with cases wherein all variables are factorial.

One source I read suggests that for these situations, statistics like the Nagelkerke psuedo-R2 because the the proportion of total variance is "less conceptually clear".

I would assume that one could make the same case for AIC.

So my overall question then, is that if residuals are the key element in determining goodness of fit statistics for regression models, but i they are not as relevant in logistic regression models, how does one evaluate the goodness of fit in a model with purely factorial variables?

Answer 1

If a binary logistic regression model is saturated, then the residuals are completely uninteresting. That's a step beyond just having factor predictors; it's basically having factor predictors will all interactions present.

When the model is saturated, the data can be divided up into groups by combinations of predictors so that every observation in a group has the same predictors and the fitted mean. Suppose there are $n$ observations in the group and the fitted mean is $\hat p$. There must necessarily be exactly $\hat p n$ observations with $Y=1$, which have residual $1-\hat p$, and exactly $(1-\hat p)n$ observations with $Y=0$, which have residual $\hat p$, so you can't learn anything more by looking at the residual distribution, and since all the observations in the group have the same predictors, you can't learn anything more by, eg, plotting residuals against predictors.

When the outcome isn't binary but is binomial then residuals aren't useless: there's more than one possible value for a positive residual or for a negative residual and you can look at the distribution. You can also assess whether the distribution of $Y$ is binomial or, say, overdispersed.

When the model isn't saturated you can use residuals to help decide what interactions should be in the model.

Answer 2

The short answer is yes, residuals are important with logistic regression using categorical features. In fact, there are entire classes of models that are fitted to this information -- widely known as LCA or latent class analysis models.

At the simplest level of understanding, LCAs are fit to the heterogeneity in the vector of residuals by partitioning the unstructured variance from such models into groups following classic rules of clustering which minimizes the variance within groups relative to the total variance. One of the nice features of such maximum likelihood-based models is that a formal test of the correct number of groupings is possible.

LCAs originated in the post-WWII quantitative explosion with the work of Columbia sociologist Paul Lazarsfeld. Other mathematical sociologists who continued to develop variants of his model include James Coleman (Lazarsfeld's student), Clifford Clogg and Leo Goodman. Coleman's 1990 book Foundations of Social Theory used econometric models based in LCA. Clogg developed software known as MLLSA which was the best off-the-shelf LCA program up through the 80s. Goodman, another U of Chicago prof like Coleman, developed R-C (or Row-Column) models for contingency table analysis.

Today, the best software for traditional LCA is prolly Statistical Innovations' Latent Gold. While it's inexpensively proprietary, they also have a ton of free white papers on their website which illuminate and leverage LCA.

Traditional LCA is a 20th c tool developed on small datasets. As such, it has been eclipsed by the work of Bayesian machine learning enthusiasts such as David Dunson at Duke who has papers on next-gen, LCA-like regressions based on tensor factorization, e.g., https://arxiv.org/abs/1509.06490