I am fitting regression models using closely related and highly collinear critical variables.

I have used a few collinearity-reduction techniques, such as:

  • PCA,
  • variable selection.

Nevertheless, there is scope for further reduction.

Since my primary goal is to perform do inference on the variables, dimensionality reduction and variable selection are not the best options for me, because they make the variables and the models harder to interpret or even outright uninterpretable. Sadly, regularization is not an option for me given the specific type of regression model and the specific software package I am using.

Are there any other collinearity reduction methods that preserve as much information as possible while keeping the information contained in each of the critical variables distinct?

  • $\begingroup$ How would scaling have any effect whatsoever on collinearity? $\endgroup$ – Tim Nov 21 '17 at 22:16
  • $\begingroup$ There might be people on this forum who understand the reasons better than I do. There is a big reduction in the kappa value when I scale my variables. $\endgroup$ – Des Grieux Nov 21 '17 at 22:31
  • $\begingroup$ See this post. stats.stackexchange.com/questions/16710/… $\endgroup$ – Des Grieux Nov 21 '17 at 22:36
  • $\begingroup$ See also this post researchgate.net/post/… $\endgroup$ – Des Grieux Nov 21 '17 at 22:39
  • 1
    $\begingroup$ I moved what I said in the comments to the answer I gave for the thread you linked, see here for more details on standardization and collinearity/correlation: stats.stackexchange.com/questions/16710/… $\endgroup$ – Tim Nov 23 '17 at 10:49

I am not sure what you mean by "collinearity reduction". Scaling will not affect it. If you want to keep your original variables, then the usual methods of dealing with collinearity are to used some form of biased regression.

Ridge regression is, I think, the most usual tool for dealing with colinearity while keeping the original variables. It doesn't lower collinearity, per se (nothing can do that and keep the original variables) but it reduces the inflation of variance of the estimators that is one of the main negative consequences of collinearity.

Another option, of course, is to drop some variables. If the collinearity consists of a relation between two variables, then you could drop either. If it consists in one variable being close to a linear combination of others, then you might want to drop the single variable.

  • $\begingroup$ The problem with coefficient regularization methods such as Lasso and Ridge regression is that currently there are no R packages that implement Lasso and Ridge penalties into mixed-effects regression with more than 1 random effect. $\endgroup$ – Des Grieux Nov 22 '17 at 2:50
  • $\begingroup$ By collinearity reduction methods I am referring to techniques that distribute and extract information that is (to the extent that this is plausible given the data) uniquely explained by a given variable. $\endgroup$ – Des Grieux Nov 22 '17 at 2:53
  • $\begingroup$ In that case, you will have to drop some variables. $\endgroup$ – Peter Flom Nov 22 '17 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.