# Working with systems with Perfect Multicollinearity

I am working with a time-series dataset that is based on demand-supply dynamics with several variables. THe sample data for one time period is:

> data
# A tibble: 19 x 6
Pred_error_A Pred_error_B  Pred_error_C  Pred_error_D  Pred_error_E  Pred_error_Target
<dbl>         <dbl>          <dbl>      <dbl>      <dbl>           <dbl>
1            -0.45           1.72          -0.170      1.85       -0.01          -0.7
2            -0.45           1.72          -0.170      2.35       -0.01          -1.2
3            -0.45           1.92          -0.170      2.35       -0.01          -1
4            -0.0500         0.37          -0.12       1.65       -0.01          -1.5
5             0             -0.840         -0.12       0.950      -0.01          -1.7
6             0             -0.200         -0.12       1.35       -0.01          -1.5
7             0             -0.460         -0.12       1.35       -0.01          -1.8
8             0             -0.520         -0.12       1.05       -0.01          -1.5
9             0             -0.340         -0.12       1.05       -0.01          -1.4
10             0             -0.340         -0.12       1.35       -0.01          -1.7
11             0             -0.110         -0.12       1.05       -0.01          -1.2
12             0             -0.110         -0.12       0.850      -0.01          -1
13             0              0             -0.12       0.350      -0.01          -0.4
14             0              0             -0.12      -0.15       -0.01           0.100
15             0              0             -0.12      -0.350      -0.01           0.300
16             0              0             -0.02       0          -0.01          -0.1
17             0              0              0          0           0              0
18             0              0              0          0           0              0
19             0              0              0          0           0              0


I was working on a regression for each time period with data and wanted to find the relative importance of regressors in explaining the target variable ( Pred_error_A through E and dependent variable is Pred_error_Target ) using relaimpo package in R. However, I soon realized I can't use regression setting because of near to perfect collinearity and hence no relative importance values. Is there a way to go around this? Or, can I use any other method (instead of regression)? Any suggestions will be useful.

The predictors are near to perfect and each one is important. I can't exclude any single one and I need the relative importance of all 5 regressors.

I am also trying XGBoost as an alternative. However, I am not sure if it takes care of multicollinearity.

• Would you mind to try a random forest or xgb importance analysis. Might be good to analyze variables and use PCA on this correlated ones if they have quite same meaning. Aug 14, 2020 at 20:08
• Perfect, or near-to-perfect? What can you say about the predictors that are contributing to the collinearity? Without more information it's hard to give a useful answer; the more specific what you provide, the more useful the answer is likely to be. Please provide that information by editing the question, as comments can sometimes get lost.
– EdM
Aug 14, 2020 at 21:20
• @EdM I have made the changes. Sorry for the inconvenience. Aug 14, 2020 at 21:28
• @polkas Can you suggest what objective function should I use with xgboost? I can't use linear regression or binary logistic Aug 14, 2020 at 22:57
• Principal component regression might help. Aug 15, 2020 at 13:08

All predictors and the dependent variable sum to 0.

That necessarily imposes a linear dependence, hence the near-perfect collinearity.

If you "can't eliminate even a single variable," you are forced to admit a serious limit in your ability to assign unique predictor importances. Yes, each of the individual variables might have a plausible, potentially causal relationship with outcome. But with a linear dependence you can't think of all the predictors separately. In that case the effects of any one predictor can always be expressed, at least roughly, in terms of apparent effects of the others.

In your sample data there isn't perfect multicollinearity, although variance inflation factors are fairly high (from 3.3 to 19). To see what's going on, take a simple example with 3 predictors $$A$$, $$B$$, and $$C$$, constrained by $$A+B+C=0$$.* The software won't let you fit all 3 with this perfect multicollinearity, so you fit a model with just $$A$$ and $$B$$:

$$Y=\beta_0 + \beta_A A + \beta_B B + \epsilon.$$

But you can just substitute the constraint to get an equally valid:

$$Y=\beta_0 - \beta_A C + (\beta_B-\beta_A) B + \epsilon.$$

So is the real "importance" of $$B$$ represented by $$\beta_B$$ or by $$(\beta_B -\beta_A)$$? Is it $$A$$ that is "important" or is it $$C$$?

The argument is similar, if not so dramatic, when the collinearity isn't perfect. Attempts to estimate individual predictor importance among correlated predictors are, at best, hard to interpret and can be downright misleading.

A tree-based method like xgboost doesn't get around the problem, it just might hide it better. See this discussion for example. At any branch point the software might choose one of several correlated predictors, so that they all show up in a final estimate of variable importance. But the estimated variable importances of the correlated variables will necessarily be correlated among themselves, as you could see by comparing xgboost results with different random seeds or with resampled data. That's the same problem as the variance inflation of the linear regression coefficients.

If you want to include all of a set of correlated predictors in your model, you must accept this difficulty with assigning separate importance to each of them. I would recommend seeing what others in your field have done to deal with such situations. Although this type of work is far out of my field, it seems that what you are doing bears some relationship to Leontief input-output models, for which I would assume there is a whole technology developed for time-series analysis.

*Your statement suggests that it's the sum of all the predictors plus the outcome that equals 0. In that case you need to ask why all of your predictor regression coefficients aren't coming out to the obvious values of -1 each, with an intercept of 0. Even if you are getting regression coefficients other than those, as you do with the sample data you provide (intercept is 0, but coefficients range from -1 to +11), arguments similar to this will hold. My suspicion is that your model is inherently unstable, essentially trying to model the minor unaccounted-for items that prevent the exact realization of your constraint.

• Thanks. But I can't eliminate even a single variable. I was trying to use XGBoost. I have read it's immune to multicollinearity. I wanted to know if using the reg:squared error as the objective function with xgb tree takes care of the situation? Aug 14, 2020 at 23:18
• @rg03775 please edit the question to describe in more detail the specific type of time-series regression you are trying to run and the reason why you "can't eliminate even a single variable." With more detail it might be possible to find a way that accomplishes what you want while working around the multicollinearity problem. For example, with compositional data all the component fractions are restricted to adding up to 1 and there are methods for analyzing such data.
– EdM
Aug 14, 2020 at 23:29
• Thanks, I have added a sample data and made some edits to the question. Aug 14, 2020 at 23:40