Assessing multicollinearity of dichotomous predictor variables

Question

I'm working on a project where we observe behaviour on a task (eg. response time) and model this behaviour as a function of several experimentally manipulated variables as well as several observed variable (participant sex, participant IQ, responses on a follow-up questionnaire). I don't have concerns about multicollinearity amongst the experimental variables because they were specifically manipulated to be independent, but I am concerned about the observed variables. However, I'm unsure how to assess independence amongst the observed variables, partially because I seem to get somewhat different results depending on how I set up the assessent, and also because I'm not very familiar with correlation in the context where one or both variables are dichotomous.

For example, here are two different approaches to determining if sex is independent of IQ. I'm not a fan of null hypothesis significance testing, so in both approaches I build two models, one with a relationship and one without, then compute and AIC-corrected log likelihood ratio:

m1 = lm(IQ ~ 1)
m2 = lm(IQ ~ sex)
LLR1 = AIC(m1)-AIC(m2)

m3 = glm(sex~1,family='binomial')
m4 = glm(sex~IQ,family='binomial')
LLR2 = AIC(m3)-AIC(m4)

However, these approaches yield somewhat different answers; LLR1 is about 7, suggesting strong evidence in favor of a relationship, while LLR2 is about 0.3, suggesting very weak evidence in favor of a relationship.

Furthermore, if I attempt to assess the independence between sex and another dichotomous observed variable, "yn", the resultant LLR similarly depends on whether I set up the models to predict sex from yn, or to predict yn from sex.

Any suggestions on why these differences are arising and how to most reasonably proceed?

Answer 1

I think you are trying to interpret P(A|B) and P(B|A) as if they should be the same thing. There is no reason for them to be equal, because of the product rule:

$$P(AB)=P(A|B)P(B)=P(B|A)P(A)$$

unless $P(B)=P(A)$ then $P(A|B) \neq P(B|A)$ in general. This explains the difference in the "yn" case. Unless you have a "balanced" table (row totals equal to column totals), the conditional probabilities (row and column) will not be equal.

A test for "logical/statistical independence" (but not causal independence) between categorical variables can be given as:

$$T=\sum_{ij} O_{ij} log\Big(\frac{O_{ij}}{E_{ij}}\Big)$$

Where $ij$ indexes the cells of the table (so in your example, $ij=11,12,21,22$). $O_{ij}$ is the observed value in the table, and $E_{ij}$ is what is "expected" under independence, which is simply the product of the marginals $$E_{ij}=O_{\bullet \bullet}\frac{O_{i \bullet}}{O_{\bullet \bullet}}\frac{O_{\bullet j}}{O_{\bullet \bullet}} =\frac{O_{i \bullet}O_{\bullet j}}{O_{\bullet \bullet}}$$

Where a "$\bullet$" indicates that you sum over that index. You can show that if you had a prior log-odds value for independence of $L_{I}$ then the posterior log-odds is $L_{I}-T$. The alternative hypothesis is $E_{ij}=O_{ij}$ (i.e. no simplification, no independence), for which $T=0$. Thus T says "how strongly" the data support non-independence, within the class of multinomial distributions. The good thing about this test is that it works for all $E_{ij}>0$, so you don't have to worry about a "sparse" table. This test will still give sensible results.

For the regressions, this is telling you that the average IQ value is different between the two values of sex, although I don't know the scale of the AIC difference (is this "big"?).

I'm not sure how appropriate the AIC is to a binomial GLM. It may be a better idea to look at the ANOVA and deviance tables for the LM and GLM respectively.

Also, have you plotted the data? always plot the data!!! this will be able to tell you things that the test does not. How different do the IQs look when plotted by sex? how different do the sexes look when plotted by IQ?

Answer 2

Why are you worried about multicolinearity? The only reason that we need this assumption in regression is to ensure that we get unique estimates. Multicolinearity only matters for estimation when it is perfect---when one variable is an exact linear combination of the others.

If your experimentally-manipulated variables were randomly assigned, then their correlations with the observed predictors as well as unobserved factors should be (roughly) 0; it is this assumption that helps you get unbiased estimates.

That said, non-perfect multicolinearity can make your standard errors larger, but only on those variables that experience the multicolinearity issue. In your context, the standard errors of the coefficients on your experimental variables should not be impacted.