# Questions tagged [multicollinearity]

Situation when there is strong linear relationship among predictor variables, so that their correlation matrix becomes (almost) singular. This "ill condition" makes it hard to determine the unique role each of the predictors is playing: estimation problems arise and standard errors are increased. Bivariately very high correlated predictors are one example of multicollinearity.

1,089 questions
Filter by
Sorted by
Tagged with
29 views

### How does perfect multicollinearity affect $R^2$ and $R_{\text{adj}}^2$?

I'd like to know how does perfect collinearity affect measures of fit (R squared and R squared adjusted). A mathematical approach is not necessary, just the general intuition is fine.
1 vote
25 views

### Is there a nonlinear counterpart of multicollinearity? [closed]

Is there a nonlinear counterpart of multicolinearity. i.e. if two variables x1 and x2 are nonlinearly dependent on eachother in such a way that makes interpretation the coefficients of a specific ...
9 views

### Multicollinearity and factor analysis

I’m carrying out and principal axis factor analysis to validate an existing questionnaire in a different population. The determinant of the R-matrix is incredibly low and way below the threshold 0....
1 vote
38 views

### How to measure marginal effect of interdependent variables on a binary outcome?

My data represents observations on a possible sequences of events that may lead to a positive outcome (y). Each event (A, B, C, D) is dependent on the previous event; for D to occur C must occur, for ...
27 views

### Multicollinearity in polynomial regression in R

I fit a model to my data with the following formula after a stepwise selection (x1 to x5 stand for my variables): lm(formula = outcome ~ x1+ x3 + x4 + x2+ x5+ poly(x4, 2) + poly(x2, 2) + poly(x5, 2) + ...
19 views

### GMM clustering with binary and multicollinear data

I am using GMM clustering on bank data. The data have both categorical and numerical attributes. The categorical data were converted to numerical using binary encoding. I have a couple of questions: ...
23 views

### interaction between (possible) correlated variables

Suppose I have a set of variables a,b,c,d,f,g,h,i, and I created a composite score by summing these eight variables into a composite score e=(a+b+c+d+f+g+h+i). Now I want to explore the interaction on ...
48 views

I have species abundances with associations between environmental variables. I realize the RDA will only be able to tell you the strength of the relationship of all the species abundances with the ...
• 471
1 vote
49 views

### Multicolinearity ONLY raises the variance of the coefficient estimates?

A hypothetical conversation: Person A: I am building a forecasting model. It is a logistic regression. All coefficients are statistically significant at the 5% level. I calculated the VIF for the set ...
• 3,180
14 views

### Assessing combined effects of binary predictor variables

I'm looking to analyse the effect of 20-30 binary predictor variables on a continuous response variable. I'm no statistician, so first off I'm not sure whether a regression with this many binary ...
29 views

### How to adjust regression for colinearity within certain levels of categorical variable?

I'm analyzing results from a large workforce survey, with approximately 110,000 responses to each question. We are interested in how certain demographic variables impact employees' opinions, so we ...
39 views

### Interpreting Logistic Regression Coefficients Under Collinearity

I thought this would be an easy question to find an answer to, but for the life of me I am having trouble finding anything that fully addresses my current problem: Consider a situation where we are ...
19 views

### Multicollinearity: not by some measures (VIF, TOL), yes by others (Wi, Fi) - does this invalidate the regression model?

Data is from http://peopleanalytics-regression-book.org/data/sociological_data.csv After reading this into R as sociological_data, here is my code: ...
• 541
53 views

### Multicollinearity iff determinant correlation matrix = 0

I'm studying Linear Models again, after finishing my degree some years ago. I found in my old notes that, according to my professor, one can check multicollinearity calculating the determinant of the ...
61 views

### Why do I find that OLS linear regression is robust against colinearity?

As per the textbook, OLS should fail when using colinear covariates. On their LinearRegression() documentation, sklearn states: When features are correlated and the columns of the design matrix have ...
• 105
1 vote
46 views

### Is the non-multicollinearity assumption for OLS multiple regression just an assumption of convenience?

The four assumptions for bivariate regression are:     • (L)inearity     • (I)ndepdent observations     • (N)ormal errors     • (E)qual variance And for multiple regression we add a fifth assumption:  ...
• 3,863
15 views

### Coefficients interpretation after recovering raw coefficients from a regression which used orthogonal regressors

I have an unbalanced panel of 747 observations and 15 years. After testing for Pooled, FE and RE, FE is the "best" model. However, I have multicollinearity problems. I can either remove one ...
29 views

### If a regression problem is ill-conditioned, does that mean we cannot perform SGD? What happens if we do?

By ill-conditioned regression problem, I mean that the feature matrix $X$ is not full rank. For example, X contains two or more columns highly correlated. If that's the case, $X^T\cdot X$ is not ...
• 143
1 vote
14 views

### When do control variables increase precision?

Suppose we're interested in the effect $\beta$ of a treatment $D$. To increase the precision of our estimate (ie., reduce the variance of $\hat{\beta}$), we can include a control variable $X$ that ...
• 31
1 vote
30 views

### Multicollinearity with fixed effects

I have a dataset with N=90,000 observations, where each observation is a respondent in a survey. Respondents vary across multiple countries (50) and years (8). Each respondent was only surveyed once, ...
• 11
19 views

### Should I remove Collinear variables for Random Forests

I am aware that Random Forests aren’t typically affected by collinearity issues, but I am trying to reduce how many variables I am utilising in my RF model. There are variables that are obviously ...
24 views

### "Residualize out" vs. "regress out" vs. "covary out"

I have heard the term "residualize out". I believe residualizing X out from Y means to regress Y ~ X and use the residuals in place of the original Y. How does this compare to "...
• 21
95 views

### How to interpret concurvity and wiggliness induced by low or high K knots in GAM models

Two questions: 1. Concurvity for factors =1, is this normal? 2. How do you interpret a partial effect when the effect is linear and lines for confidence intervals are pinched at 0? Is it okay for ...
• 17
74 views

### Proof that multicollinearity doesn't produce biased estimators

I'm trying to prove that multicollinearity doesn't introduce bias into a multiple linear regression model, but my proof seems to indicate the opposite. If we represent the model as y = \hat \beta_0 +...
23 views

### Does the condition number depend on the sample size?

I am working with a dataset of 214 thousand firms and I am running a logistic regression. Not only my dependent variable (bOptingOut), but also some of my independent variables are binary (...
1 vote
24 views

### Does multicollinearity affect EDA?

I have been working on a dataset pertaining to 'churn analysis'. I have been trying to demonstrate whether the customers that are being charged more are also the ones that churn more or not. My ...
1 vote
105 views

### MLR - Eliminating multicollinearity when predictors are transformations of others

I am applying a multiple linear regression on a data set, where some of the predictors are "transformations" of others (however, I'm not entirely sure if they are linear transformations or ...
169 views

### PCA can make VIF worse? Why is it so often sold as a multicollinearity fix?

It's the conventional wisdom that a PCA transformation can cure multicollinearity. Putting this into practice on example data, I find myself confused. In the following case, applying PCA seems to have ...
115 views

### Is there a difference between perfect collinearity and multicollinearity?

I've read that for multiple regression analysis there is an assumption of no perfect collinearity. Is that the same as multicollinearity?
41 views

### VIFs in Fixed-Effects-model

I know that R is providing some Warnings messages when you try to get your VIFs to test for multicollinearity. But why is that? Does someone know a good paper or the answer itself? CODE: ...
17 views

### Collinearity and relationship between the parameter estimates [duplicate]

Given a dataset with a dependent variable (Y), and three independent variables/features {X1,X2,X3}, where collinearity exists, and if we fit it using linear regression, is there any relationship ...
• 947
42 views

### Logistic regression - are interaction terms redundant vs original features if using L1 penalization for feature selection?

I am running lasso/elastic regression for feature selection in a logistic classifier. I have two continuous features, and was wondering if it would be redundant to include an interaction term or other ...
27 views

### Regress out covariates in a different order

I got a list of covariates in the multiple linear regression Y = X + cov1 + cov2 + ... + cov10 One of the covariates (e.g. cov1) is highly correlated with Y (Pearson's correlation = ~0.6). I was ...
822 views

### Is this the correct way to estimate a Difference-in-Differences model in R (multiple periods and states)?

First, I would like to say that I'm not sure if this is a question for CrossValidated or StackOverflow, as the question relates both to coding and statistics. My Q: I would like to know if this is the ...
• 145
1 vote
93 views

### Do you have to remove perfectly collinear independent variables prior to Cox regression?

Suppose you have independent variables that change only with each new time step (and possibly others that change freely): ...
63 views

### Which threshold should I use for GVIF1/(2⋅df)? (Variance Inflation Factor)

I'm using the mtcars dataset in R, I used the car packages to estimate the VIF, but since I have factor variables I got the vif table with GVIF and GVIF1/(2⋅df) values, in another question Which ...
2k views

### Why is dimensionality reduction used if it almost always reduces the explained variation?

Let's say I have $N$ covariates in my regression model, and they explain 95% of the variation of the target set, i.e. $r^2=0.95$. If there are multicollinearity between these covariates and PCA is ...
• 581
197 views

### When adding polynomial features, the issue of multicollinearity doesn't hold. Why?

In the regression model, sometimes to capture the non-linear relationship between dependent and independent variables, we use polynomial features. But in regression, if the two or more features are ...
67 views

• 91
1 vote
29 views

### Is there any way to determine VIF of some variable that is included in the dataset that has so many variables?

I am new in statistics and need some help to determine the VIF value on all my variables/features in the dataset with a lot of variables. I have 98 variables with 76 observations and need to find VIF ...
• 11
53 views

### Polynomial regression and multicollinearity [duplicate]

to use linear regression we should have the predictors/features be independent from one another. However, in polynomial regression, x^2 is perfectly correlated to x ( knowing x value gives us the ...
52 views

### Multicollinearity for Logistic Regression and Neural Network

I am looking to fit Logistic Regression (LR) and Neural Networks (NN) models in order to predict if there will be avalanches during a day (0 or 1 dependant variable) based on meteorological variables (...
• 31
1 vote
32 views

### How to handle high multicollinearity for logistic regression and neural nets [closed]

I am looking to fit multiple models (Random Forest (RF), Classification Trees (CT), Logistic Regression (LR) and Neural Networks (NN)) in order to predict if there will be avalanches during a day (0 ...
• 31