I am using the statsmodel.ols module to compute an omnibus (ANOVA) F-test for three within-subjects factors; 2*3*2 levels design. The Cond. No. of the omnibus test (26.2) suggests multicollinearity. My understanding is that this means the model parameters are correlated, is that a correct statement? And a follow-up, what is the appropriate remedy in this case? E.g., Non-parametric alternative or some sort of adjustment.

Edit: The question here concerns electrophysiological time series data for repeated measurements. I have 3 independent categorical variables (x1, x2, x3) with 3x2x2 levels respectively. Briefly, I have specific hypotheses about whether auditory stimulation (3 distinct types) and socioeconomic status level (low vs. high) modulate neural activity recorded at the scalp from sensor arrays over the two hemispheres (left vs. right) in the brain.

The initial omnibus ANOVA indicated a significant interaction between factors (stimulusseshemisphere), and also multicollinearity on the basis of the Cond. No. metric in statsmodel.ols. Prior to this question I was unaware of VIF, thus this question does overlap with Multicollinearity when individual regressions are significant, but VIFs are low, though I am unsure about the VIFs in this case.

  • $\begingroup$ Possible duplicate of Multicollinearity when individual regressions are significant, but VIFs are low $\endgroup$ – Carl Oct 23 '18 at 0:19
  • $\begingroup$ I think we need more details here about the underlying design and scientific question. If this is a designed experiment then how did you arrive at such an unbalanced design? Are you trying to explain the effects of your unnamed variables you just predict your outcome? $\endgroup$ – mdewey Oct 23 '18 at 8:46
  • $\begingroup$ @mdewey for my own education, how do you go about deciding whether the design is balanced or not? $\endgroup$ – Kambiz Oct 26 '18 at 6:26
  • $\begingroup$ The design is unbalanced if you have unequal numbers in the cells. In your case the cells are formed by the 2*3*2 factor combinations. If those parameters are correlated I assumed it was because your design was unbalanced and in a designed experiment that seemed hard to understand. If it is balanced then we need further details I think. $\endgroup$ – mdewey Oct 26 '18 at 10:52
  • $\begingroup$ How many study participants did you have? $\endgroup$ – mdewey Oct 26 '18 at 12:15

First, multicolinearity indicates that there is a linear relationship among your independent variables. Correlation is neither a necessary nor a sufficient condition for collinearity (although, with only 3 IVs, it is very hard to have one without the other - with more IVs, it is entirely possible).

Second, if you are deciding between ridge and lasso, I would go with ridge regression here. See this thread for some notes on ridge regression with categorical variables. Ridge regression produces biased parameter estimates in order to reduce the variance of the estimates. It won't (usually) remove variables entirely. Lasso removes some variables from the equation and that probably isn't what you want here, especially if the interaction is important.

Third, I think partial least squares is a better solution to collinearity than principal components, because PLS also considers the relationship with the dependent variable. However, with only three independent variables, you are likely to get a single component and I think it is unlikely that that will give you a useful result. Also, see this thread for some notes on PLS with categorical variables.

Finally, have you considered regression trees and their offshoots such as random forests?


Multicollinearity means that some of your explanatory variables are correlated. Multicollinearity is a problem because it inflates the variance of your beta estimates, which could cause your beta estimates to be statistically indifferent from 0 (which could also affect your F-test).

If you feel that you are justified in including all of the explanatory variables, you could try a regression that reduces the variance (at the cost of biasing your estimates [I highly suggest you read up on the bias-variance trade-off]). This includes a ridge regression or a lasso regression (they work similarly).

The usual remedy to multicollinearity is an application of Principal Components Analysis to your data (which would re-map your explanatory variables, preserve most of the variation in your data, and completely eliminate multicollinearity). This is not the remedy in the case of data coming from experiments, as you want to avoid re-mapping your data (because you lose interpretability).

  • 1
    $\begingroup$ +1 Thank you. My research into the matter had me leaning towards a partial least squares. But I have the impression that may be too similar to PCA in its treatment of the data, do you agree? I'll take a look at the material you suggest. $\endgroup$ – Kambiz Oct 22 '18 at 19:26
  • $\begingroup$ Partial Least Squares is another solution, however, it suffers from the same flaws as the PCA->OLS method because you're remapping your data. Definitely consider Lasso. $\endgroup$ – OUrista Oct 22 '18 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.