How to prevent collinearity?

Question

Ieno & Zuur 2015 describe a number of causes of collinearity among explanatory variables entered into a linear regression. One of these causes is what they call a 'data collection' cause.

They say that the data collection cause of collinearity can occur in the following scenario:

You go into the field, follow a track, and along the track you count, e.g. monkeys and various explanatory variables. If the track goes uphill, the explanatory variable slope will change, as will the type of trees, the type the type of vegetation, the tree volume, etc.

They go on to say:

The solution is to choose a sampling design that avoids such problems.

What sampling designs will avoid collinearity problems?

Answer 1

As far as I understand, collinearity or multicollinearity (hereafter referred to simply as collinearity) cannot be prevented/avoided during data analysis, because collinearity is a built-in "feature" of data. Therefore, a particular data set has certain levels of collinearity (or the lack of). However, collinearity can be prevented/avoided to some degree prior to data analysis, that is, during research design planning or, possibly, exploratory data analysis (EDA) phases. This is likely what Ieno and Zuur (2015) mean by their phrase, which you've cited in your question above.

Potential solutions for preventing / avoiding / dealing with collinearity include using appropriate research designs, which reduce collinearity. However, while I ran across mentioning this approach several times, it was unclear to me which designs exactly are helpful in that regard and why (while StatsStudent mentions one such method - stratified sampling, relevant sources are not provided). Before mentioning other solutions, it is worth to say that sometimes recommended option of dropping predictors is considered as rather bad one - see this blog post or the blog author's book (Baguley, 2012). He also mentions that doing nothing should be considered as one of the valid approaches to dealing with collinearity as well.

Other approaches to dealing with (mainly reducing) collinearity include: increasing sample size and transforming predictors (Baguley, 2012); using principal component analysis (PCA), using simple regression between highly correlated variables (sequential regression) and calculating ratio of correlated variables (Balling, n.d.); a priori modeling and ridge regression (Graham, 2003). While much of literature is focused on dealing with collinearity in multiple regression settings, it should be noted that researchers, who use structural equation modeling (SEM) in their studies, face similar issues of collinearity (Grewal, Cote & Baumgartner, 2004). This is despite the fact that latent variable modeling (LVM) is also considered as an approach to reducing collinearity (see below).

Finally, I highly recommend a very comprehensive paper on the topic by Dormann, Elith, Bacher, Buchmann, Carl, Carré et al. (2013), which contains an excellent overview of methods for dealing with collinearity as well as their comparison via simulation. Those methods include: PCA and other variables clustering methods; already mentioned sequential regression; principal component regression (PCR), partial least squares (PLS) and some other LVM methods; tolerant/penalized regression techniques, include the above-mentioned ridge regression.

References

Baguley, T. (2012). Serious stats: A guide to advanced statistics for the behavioral sciences. New York, NY: Palgrave Macmillan.

Balling L. W. (n.d.). A brief introduction to regression designs and mixed-effects modelling by a recent convert. Retrieved from http://pure.au.dk/portal/files/14325917/balling_csl.pdf

Dormann, C. F., Elith, J., Bacher, S., Buchmann, C., Carl, G., Carré, G., ..., & Lautenbach, S. (2013). Collinearity: A review of methods to deal with it and a simulation study evaluating their performance. Ecography, 36(1), 27-46. doi:10.1111/j.1600-0587.2012.07348.x Retrieved from http://onlinelibrary.wiley.com/doi/10.1111/j.1600-0587.2012.07348.x/pdf

Graham, M. H. (2003). Confronting multicollinearity in ecological multiple regression, Ecology, 84(11), 2809-2815. Retrieved from http://www.auburn.edu/~tds0009/Articles/Graham%202003.pdf

Grewal, R., Cote J. A., & Baumgartner, H. (2004). Multicollinearity and measurement error in structural equation models: Implications for theory testing. Marketing Science, 23(4), 519-529. doi:10.1287/mksc.1040.0070 Retrieved from http://www.personal.psu.edu/rug2/Grewal,%20Cote,%20%26%20Baumgartner%20MKS%202004.pdf

Answer 2

There are several sampling techniques that can be used to reduce collinearity, so I'll mention just one of them: a stratified random sampling plan can eliminate some of the problem. Here's an example using the scenario you described above: If you find that certain trees are collinear with altitude or slope of the track, you could stratify the population-area on categories (areas) of altitude or steepness (e.g. form a stratum based on the following bins: 0-5 slope, 5-10 slope, 10-15 slope. . . ). By creating slope strata, you can avoid reduce some of the the issue of multicollinearity with slope.

Answer 3

Obviously, I can't speak for Leno or Zuur. I am also not an ecologist. However, from the description, I think they just mean walk more than one path. In essence, multicollinearity just means that your variables are correlated with each other.

If you walk only one path up the mountain, every variable you measure at each point will be at least somewhat related by virtue of the fact that they are measured at the same altitude. On the other hand, if you walk up the mountain on the south side, and also walk up on the north side, you may find that the variables differ even at the same altitude. For example, (depending on where you are on the planet) one side will get more sunlight than the other side. One side may get more rain. Etc. By walking more, and more dissimilar, paths, you can minimize the collinearity that would otherwise have arisen.

Answer 4

Similar to @StatsStudent,

You need replication of the other variables within values of the same slope. E.g. be also doing transcets around the mountain (sideways) starting at different elevations/slopes.

If there is truly no variaiton in tree trype with slope, then you will never be able to seperate the two using natural variation as it is non-existant. Unless you bin data within different ranges of slope (but this is not necessary a sampling thing as you can do this afterwards during data analysis..) In this case you would need an EXPERIMENT :)

Answer 5

The example you mention is a little overly simplified for my taste. But the sentiment is right: correcting for collinearity is not a post-hoc procedure. This is guaranteed to lead to data dredging, since the result of collinearity is variance inflation, and adding or removing factors to get significant results (i.e. from reduced variance) is not a statistically sound procedure.

To dive into the example more, I think we have to distinguish randomized experimental studies and pseudo-randomized or observational studies.

randomized approaches

Basically, the examples suggests in both scenarios: use experimental or pseudo-experimental design to reduce collinearity of factors. Randomization should eliminate collinearity. This is called the randomization assumption. Blocking or blocked randomization, or blocked design, ensures that covariates are balanced exactly rather than probabilistically (i.e. in their expected value). Similar to this is re-randomization; you can re-roll your die to obtain covariate levels which are balanced if the cohort design is not staggered. Other blocking strategies have been discussed elsewhere, like Latin Squares, and so on.

pseudo-randomized approaches

In observational studies, then, a new field of causal modeling has emphasized the rationale for adjusting for certain variables in the analysis. With regards to whether it causes collinearity or not, we say, "So what?" And include it anyway. For instance, social depravity is highly correlated with the likelihood of child smoking and it is also highly correlated with children's health. When you adjust for its influence in measures of association between smoking and health, you find a somewhat attenuated estimate. This is because, by not accounting for social depravity, smoking is to some extent a measure of the hardship of life: low income, stable housing, adequate food, and so on, as well as the direct effect of smoking on health.

Often studies aren't randomized, so that several confounding variables may affect the association between a pseudo-treatment and an outcome. Here, a pseudo-treatment is something that can't be randomized for one reason or another, like smoking in children. Matching or paired design based on exposed and unexposed individuals in strata defined by confounders would result in the confounders being orthogonal to the exposure. The distribution of confounders then is balanced between exposed and unexposed participants (you may also categorize the exposure into quantiles or such and obtained matched samples there).

Related to this, you might consider using propensity matching. The concept was born out of an approach to blend observational and experimental design. It turns out, if you can construct a prediction model for the likelihood of receipt of exposure, based on measured confounding variables, that model produces a propensity score. By matching equally likely cases and controls, you may obtain a subsample of the observational sample which is "balanced" in some sense. But again, choice of confounders post hoc is often a bad idea, because we rarely just "conveniently" measure them, they must be pre-specified. Omitting confounders results in poor performance and bias of propensity matched analyses.

However, both of these approaches are equivalent to just adjusting for the confounders in the model.

So in both cases, collinearity is something which is addressed a priori to data collection. The question, then, shouldn't be how to prevent collinearity from a data analysis perspective. It's simply to make a note of it, and account for the reduction in power. If a factor is the correct thing to adjust for in a model, you must do it if you can. If it renders results non-significant when an association would be expected, you may hypothesize that variance inflation overspent your power, and additionally report unadjusted associations, commenting on their differences. This would be an inconclusive finding, however.

Answer 6

Generally your sample should be a random sample, therefore what you read is the intuitive description of credible randomness. The opposite would be systematic bias like are you only sampling from one town in a country? Or from one family in a village? Or is there a strong time trend and your cross-section is a quasi time series you're ignoring? Ask yourself these questions and if the answer is no then there should be no multicollinearity in the data.

Another issue is this: most estimation procedures involve inverting a second moment matrix of the type $(X'X)^{-1}$. Now, if your data quality is so low that there is not a lot of variation in it, and this will be reflected in the second moment matrices as they measure the spread, then they may be near non-invertible.

Your standard errors will be huge (b/c you're effectively dividing by zero) in this case meaning your t-stats will be low leading to less parameter significance but if you're stuck at the stage of inverting second moment matrices, then you can find a generalized inverse that will get you some coefficient but as I wrote, it is most likely insignificant.

Answer 7

Remember collinearity is a consequence of having multiple predictor variables that have a large correlation because.... they might have too much in common! And this creates a problem because they make the analysis of the impact on the response very hard to understand since those variables affect significantly the change on the response at the same time. Therefore, in order to avoid this to happen (and introduce noise to your model) you need to evaluate each independent variable y on x(i) on analogous scatter plots (correlation plots) to see how they independently affect the response. Also, is good to get some background of what the variables are actually describing, because they might share similar properties or attributes, so you are basically describing something twice... Also, you can use the variance inflation factor (VIF) which quantifies the amount of variance for each variable. The correlation between two variables define the VIF, so the larger the VIF the larger the standard error. VIF helps asses the proportion if variation in a variable after the other one happened. VIF = 1 (or very close) means that the collinearity effect between variables is not statistically significant.

By using the plots, and using the VIF, you can have an idea of the collinearity and eventually take the decision to keep or remove a variable to improve your model.

There are several treatments that can be applied to collinearity

• remove redundant variables
• Re-express explanatory variable (maybe a ratio?)
• do nothing, since the collinearity might not be to high but still helps to explain the change on the response.