# How to avoid collinearity of categorical variables in logistic regression?

I have the following problem: I'm performing a multiple logistic regression on several variables each of which has a nominal scale. I want to avoid multicollinearity in my regression. If the variables were continuous I could compute the variance inflation factor (VIF) and look for variables with a high VIF. If the variables were ordinarily scaled I could compute Spearman's rank correlation coefficients for several pairs of variables and compare the computed value with a certain threshold. But what do I do if the variables are just nominally scaled? One idea would be to perform a pairwise chi-square test for independence, but the different variables don't all have the same co-domains. So that would be another problem. Is there a possibility for solving this problem?

• Not a duplicate, but a similar one: stats.stackexchange.com/questions/200720/… . Don't be misled by the title, OP of that question meant independent variables. Also, see Peter Flom's answer to this question: stats.stackexchange.com/questions/72992/… – T.E.G. Jan 9 '17 at 17:32
• Why do you "want to avoid multicollinearity"? Sometimes it's inevitable, even helpful, and can be dealt with by approaches like ridge regression. If your nominal variables were individual items that together made up a Likert scale, then you would actually depend on their multicollinearity and could use the sum of such items as a predictor. So what in particular about your study makes it so necessary to avoid multicollinearity? – EdM Jan 9 '17 at 18:09

I would second @EdM's comment (+1) and suggest using a regularised regression approach.

I think that an elastic-net/ridge regression approach should allow you to deal with collinear predictors. Just be careful to normalise your feature matrix $X$ appropriately before using it, otherwise you will risk regularising each feature disproportionately (yes, I mean the $0/1$ columns, you should scale them such that each column has unit variance and mean $0$).

Clearly you would have to cross-validate your results to ensure some notion of stability. Let me also note that instability is not a huge problem because it actually suggests that there is not obvious solution/inferential result and simply interpreting the GLM procedure as "ground truth" is incoherent.

The ViF is still a useful measure in your case, but the condition number of your design matrix is a more common approach for categorical data.

The original reference is here:

Belsley, David A.; Kuh, Edwin; Welsch, Roy E. (1980). "The Condition Number". Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York: John Wiley & Sons. pp. 100–104.

And here are more useful links:
https://en.wikipedia.org/wiki/Condition_number

• +1. Hm... A bit "blast-from-the-past" that report but interesting. I suspect that this approach will be a pain when dealing with multiple variables that have multiple levels (their applications are binary or trinary categorical variables) but yeah, interesting read! – usεr11852 Jan 9 '17 at 19:19

Another approach would be to perform Multiple Correspondence Analysis (MCA) on your multicollinear independent variables. After that you will end up with orthogonal (perfectly independent) components which you can use as IV in your model. There will be no collinearity present, but it will be hard to intepret effects of your original variables. At the other hand if there is multicollinearity, MCA will unite your correlated IV variables effects into more general effects, which you can find even more interpretable and plausible.

You can check bi-variate correlation by using rank-order or other non-parametric test for categorical variables. It is the same as you check the correlation matrix for a group of continuous variables, just use different test.

• The OP has already said he has rejected this as his variables are not ordered categorical. – mdewey Jan 9 '17 at 18:21