How to eliminate variables from regression models due to collinearity and multicollinearity (linear, Poisson, and negative binomial)

Question

I am trying to eliminate variables from the regression model (linear, Poisson, and negative binomial) due to the high value of correlation. The R code is below

pairs.panels(a12, cex.cor = 4, cex.labels = 4, cex.axis = 2, method = "pearson")
pairs.panels(a12, cex.cor = 4, cex.labels = 4, cex.axis = 2, method = "spearman")
pairs.panels(a12, cex.cor = 4, cex.labels = 4, cex.axis = 2, method = "Kendall") 

Because my variables' value did not follow normal distribution so that I mainly considered values from the Spearman method. I saw a value of higher than 0.7 (ex 0.77) and some values which were higher than 0.6.

I am not sure whether my thinking for methods eliminating variables is correct.

I will do

1. I will delete variable (between two correlated variables) which had lower correlation value to my target variable (response, dependent)
2. I used AIC and BIC to find the models with best value (lowest AIC,BIC)
3. I apply LASSO, Bayesian Model Averaging (BMA package), statistically equivalent signature (SES, MXM package)
4. I also find an interesting information here: https://www.statalist.org/forums/forum/general-stata-discussion/general/650016-decide-which-variables-to-be-omitted-in-ols-regression

Thanks to comments from readers, I would like to add more information from my model:

1. Type of outcome value: I do count data regression so that expected outcome would be 0, 1, 2, 3,........
2. I have a total of 104 observations for my model
3. My model has 1 target variable and six predictors (independent variables)
4. I used the model to examine the causal relationship between the number of facilities appearing within a ward based on population, area, .... and make predictions, too.
5. I think that I should eliminate my variables because there are some high values of correlation (0.6 to 0.8) between the two variables in my model. I have read documents that this would affect the precision of the coefficient and standard error of the coefficient of the model.
6. I have checked VIF for all variables (using linear regression using car package), and all value were lower than 5

Would you have any references or comments for me? I am not sure about these above-mentioning methods

Answer 1

I think that I should eliminate my variables because there are some high values of correlation (0.6 to 0.8) between the two variables in my model. I have read documents that this would affect the precision of the coefficient and standard error of the coefficient of the model.

Although correlation affects the precision of the estimates of individual regression coefficients, it need not affect the overall model quality, predictions from the model, or "chunk" tests that evaluate the combined associations between the correlated predictors and outcome. This page illustrates for a linear regression with two correlated predictors, but the principles apply more generally.

When you have two correlated predictors, the variances of their individual coefficient estimates are increased by the variance inflation factor (VIF). But there is also a counterbalancing covariance between those estimates that is incorporated into predictions from the model and into any tests that combine the two correlated predictors.

Alternatively, you could devise a new predictor that combines the values of both the correlated predictors, for example a weighted sum that takes into account differences in scale. That's one type of data reduction that can decrease the effective number of predictors in the model, using information from all the originally observed predictor values.

I have a total of 104 observations for my model... My model has 1 target variable and six predictors (independent variables)

Cutting down on the number of predictors in the model is important when the total number of predictors is close to or greater than the number of observations. That can happen in the types of large-scale data dredging that is implicitly assumed in much of what's written about multicollinearity.

You don't have that problem. You have 17 times as many observations as predictors. Even if you need to assign multiple terms in the model to some predictors (e.g., for multi-level categorical predictors or spline fits of continuous predictors), you should be able to fit a model without much risk of the overfitting that arises from too low an observation/predictor ratio. And you can always check for overfitting by repeating the modeling process on multiple bootstrapped samples of your data.

I used the model to examine the causal relationship between the number of facilities appearing within a ward based on population, area, .... and make predictions, too.

As noted above, predictions from the model will probably be improved by including all predictors. A regression model on its own won't establish causal relationships. Causal inference is much more involved. See Causal Inference: What If? by Hernán and Robins for an accessible introduction.