Questions tagged [multicollinearity]

Situation when there is strong linear relationship among predictor variables, so that their correlation matrix becomes (almost) singular. This "ill condition" makes it hard to determine the unique role each of the predictors is playing: estimation problems arise and standard errors are increased. Bivariately very high correlated predictors are one example of multicollinearity.

Filter by
Sorted by
Tagged with
91
votes
9answers
36k views

Is there an intuitive explanation why multicollinearity is a problem in linear regression?

The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very ...
81
votes
0answers
64k views

How can a regression be significant yet all predictors be non-significant? [duplicate]

My multiple regression analysis model has a statistically significant F value however all beta values are statistically non-significant. All the regression assumptions are met. No multicollinearity ...
77
votes
9answers
53k views

Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?

In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests? In my model, there are 10 ...
69
votes
1answer
75k views

What correlation makes a matrix singular and what are implications of singularity or near-singularity?

I am doing some calculations on different matrices (mainly in logistic regression) and I commonly get the error "Matrix is singular", where I have to go back and remove the correlated variables. My ...
51
votes
3answers
97k views

What is the effect of having correlated predictors in a multiple regression model?

I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
50
votes
6answers
29k views

Why is multicollinearity not checked in modern statistics/machine learning

In traditional statistics, while building a model, we check for multicollinearity using methods such as estimates of the variance inflation factor (VIF), but in machine learning, we instead use ...
38
votes
2answers
41k views

Won't highly-correlated variables in random forest distort accuracy and feature-selection?

In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables ...
31
votes
3answers
38k views

Which variance inflation factor should I be using: $\text{GVIF}$ or $\text{GVIF}^{1/(2\cdot\text{df})}$?

I'm trying to interpret variance inflation factors using the vif function in the R package car. The function prints both a ...
29
votes
3answers
21k views

How to tell the difference between linear and non-linear regression models?

I was reading the following link on non linear regression SAS Non Linear. My understanding from reading the first section "Nonlinear Regression vs. Linear Regression" was that the equation below is ...
28
votes
3answers
38k views

How to deal with multicollinearity when performing variable selection?

I have a dataset with 9 continuous independent variables. I'm trying to select amongst these variables to fit a model to a single percentage (dependent) variable, ...
28
votes
5answers
101k views

How to test and avoid multicollinearity in mixed linear model?

I am currently running some mixed effect linear models. I am using the package "lme4" in R. My models take the form: ...
28
votes
2answers
15k views

Is PCA unstable under multicollinearity?

I know that in a regression situation, if you have a set of highly correlated variables this is usually "bad" because of the instability in the estimated coefficients (variance goes toward infinity as ...
26
votes
2answers
7k views

Collinearity diagnostics problematic only when the interaction term is included

I've run a regression on U.S. counties, and am checking for collinearity in my 'independent' variables. Belsley, Kuh, and Welsch's Regression Diagnostics suggests looking at the Condition Index and ...
23
votes
6answers
14k views

Dealing with correlated regressors

In a multiple linear regression with highly correlated regressors, what is the best strategy to use? Is it a legitimate approach to add the product of all the correlated regressors?
23
votes
1answer
5k views

Is there a reason to prefer a specific measure of multicollinearity?

When working with many input variables, we are often concerned about multicollinearity. There are a number of measures of multicollinearity that are used to detect, think about, and / or communicate ...
20
votes
3answers
49k views

How to systematically remove collinear variables in Python? [closed]

Thus far, I have removed collinear variables as part of the data preparation process by looking at correlation tables and eliminating variables that are above a certain threshold. Is there a more ...
19
votes
1answer
12k views

Logistic Regression - Multicollinearity Concerns/Pitfalls

In Logistic Regression, is there a need to be as concerned about multicollinearity as you would be in straight up OLS regression? For example, with a logistic regression, where multicollinearity ...
18
votes
1answer
54k views

How to deal with high correlation among predictors in multiple regression?

I found a reference in an article that goes like: According to Tabachnick & Fidell (1996) the independent variables with a bivariate correlation more than .70 should not be included in ...
17
votes
2answers
20k views

Qualitative variable coding in regression leads to “singularities”

I have an independent variable called "quality"; this variable has 3 modalities of response (bad quality; medium quality; high quality). I want to introduce this independent variable into my multiple ...
17
votes
3answers
47k views

When can we speak of collinearity

In linear models we need to check if a relationship exists among the explanatory variables. If they correlate too much then there is collinearity (i.e., the variables partly explain each other). I am ...
15
votes
1answer
1k views

Why does this regression NOT fail due to perfect multicollinearity, although one variable is a linear combination of others?

Today, I was playing around with a small dataset and performed a simple OLS regression which I expected to fail due to perfect multicollinearity. However, it didn't. This implies that my understanding ...
15
votes
2answers
29k views

VIF, condition Index and eigenvalues

I am currently assessing multicollinearity in my datasets. What threshold values of VIF and condition index below/above suggest a problem? VIF: I have heard that VIF $\geq 10$ is a problem. After ...
14
votes
1answer
8k views

Why does Ridge Regression work well in the presence of multicollinearity?

I am learning about ridge regression and know that ridge regression tends to work better in the presence of multicollinearity. I am wondering why this is true? Either an intuitive answer or a ...
14
votes
3answers
14k views

Do I need to drop variables that are correlated/collinear before running kmeans?

I am running kmeans to identify clusters of customers. I have approximately 100 variables to identify clusters. Each of these variables represent the % of spend by a customer on a category. So, if I ...
14
votes
4answers
5k views

Should one be concerned about multi-collinearity when using non-linear models?

Say we have a binary classification problem with mostly categorical features. We use some non-linear model (e.g. XGBoost or Random Forests) to learn it. Should one still be concerned about multi-...
13
votes
6answers
8k views

Multicollinearity when individual regressions are significant, but VIFs are low

I have 6 variables ($x_{1}...x_{6}$) that I am using to predict $y$. When performing my data analysis, I first tried a multiple linear regression. From this, only two variables were significant. ...
13
votes
3answers
5k views

How can you handle unstable $\beta$ estimates in linear regression with high multi-collinearity without throwing out variables?

Beta stability in linear regression with high multi-collinearity? Let's say in a linear regression, the variables $x_1$ and $x_2$ has high multi-collinearity (correlation is around 0.9). We are ...
13
votes
2answers
5k views

What are chunk tests?

In answer to a question on model selection in the presence of multicollinearity, Frank Harrell suggested: Put all variables in the model but do not test for the effect of one variable adjusted ...
13
votes
5answers
5k views

Does standardising independent variables reduce collinearity?

I've come across a very good text on Bayes/MCMC. IT suggests that a standardisation of your independent variables will make an MCMC (Metropolis) algorithm more efficient, but also that it may reduce (...
13
votes
2answers
19k views

Variance-Covariance matrix interpretation

Assume we have a linear model Model1 and vcov(Model1) gives the following matrix: ...
13
votes
2answers
858 views

Why doesn't collinearity affect the predictions?

I have read in many places that collinearity doesn't affect the predictions. It only affects the coefficient tests and confidence interval. As a result it cannot be used for causal inference but for ...
13
votes
2answers
4k views

Dealing with multicollinearity

I have learnt that using vif() method of car package, we can compute the degree of multicollinearity of inputs in a model. From ...
13
votes
1answer
3k views

Interpreting proportions that sum to one as independent variables in linear regression

I'm familiar with the concept of categorical variables and the respective dummy variable coding that allows us to fit one level as baseline so as to avoid collinearity. I'm also familiar with how to ...
13
votes
2answers
427 views

Linear regression when you only know $X^t Y$, not $Y$ directly

Suppose $X\beta =Y$. We don't know $Y$ exactly, only its correlation with each predictor, $X^\mathrm{t}Y$. The ordinary least-squares (OLS) solution is $\beta=(X^\mathrm{t} X)^{-1} X^\mathrm{t}Y$ ...
12
votes
3answers
7k views

What is an example of perfect multicollinearity?

What is an example of perfect collinearity in terms of the design matrix $X$? I would like an example where $\hat \beta = (X'X)^{-1}X'Y$ can't be estimated because $(X'X)$ is not invertible.
12
votes
1answer
6k views

Is Support Vector Machine sensitive to the correlation between the attributes?

I would like to train an SVM to classify cases (TRUE/FALSE) based on 20 attributes. I know that some of those attributes are highly correlated. Therefore my question is: is SVM sensitive to the ...
12
votes
2answers
18k views

Collinearity between categorical variables

There's a lot about collinearity with respect to continuous predictors but not so much that I can find on categorical predictors. I have data of this type illustrated below. The first factor is a ...
12
votes
1answer
2k views

Is there a problem with multicollinearity and for splines regression?

When using natural (i.e. restricted) cubic splines, the basis functions created are highly collinear, and when used in a regression seem to produce very high VIF (variance inflation factor) statistics,...
11
votes
2answers
1k views

How to start building a regression model when the most strongly associated predictor is binary

I have data set containing 365 observation of three variables namely pm, temp and rain. Now ...
11
votes
3answers
948 views

What are the merits of different approaches to detecting collinearity?

I want to detect whether collinearity is a problem in my OLS regression. I understand that variance inflation factors and the condition index are two commonly used measures, but am finding it ...
11
votes
5answers
17k views

What to do with collinear variables

Disclaimer: This is for a homework project. I'm trying to come up with the best model for diamond prices, depending on several variables and I seem to have a pretty good model so far. However I have ...
11
votes
4answers
4k views

Is multicollinearity really a problem?

I am working on some predictive modeling project these days: trying to learn a model and make real-time predictions based on the model I learned offline. I started using ridge regression recently, ...
11
votes
1answer
316 views

Does multicollinearity increase the variance of the beta for every covariate or just those that are collinear?

I know under typical OLS assumptions $$Var(\hat{\beta}) = [X^TX]^{-1}X^T Var(\epsilon)X[[X^TX]^{-1}]^T$$ If some $X_j$ can be approximately written as a linear cominbation of the other covariates, ...
11
votes
1answer
409 views

Reference for the sum and difference of highly correlated variables being almost uncorrelated

In a paper I've written I model the random variables $X+Y$ and $X-Y$ rather than $X$ and $Y$ to effectively remove the problems that arise when $X$ and $Y$ are highly correlated and have equal ...
10
votes
3answers
6k views

Linear relationship between explanatory variables in multiple regression

I was reading the multiple regression chapter of Data Analysis and Graphics Using R: An Example-Based Approach and was a bit confused to find out that it recommends checking for linear relationships ...
10
votes
2answers
3k views

Is multicollinearity implicit in categorical variables?

I noticed while tinkering with a multivariate regression model there was a small but noticeable multicollinearity effect, as measured by variance inflation factors, within the categories of a ...
10
votes
1answer
241 views

Standardization of variables and collinearity

Collinearity can pose certain problems in various kinds of regression problem. In particular, it can make the parameter estimates have high variance and be unstable. Various methods have been ...
10
votes
1answer
1k views

Variance inflation factor for generalized additive models

In the usual VIF calculation for a linear regression, each independent/explanatory variable $X_j$ is treated as the dependent variable in an ordinary least squares regression. i.e. $$ X_j = \beta_0 + ...
9
votes
3answers
1k views

Set of uncorrelated but linearly dependent variables

Is it possible to have a set of $K$ variables that are uncorrelated but linearly dependent? i.e. $cor(x_i, x_j)=0$ and $ \sum_{i=1}^K a_ix_i=0$ If yes can you write an example ? EDIT: From the ...
9
votes
1answer
2k views

Can standardized $\beta$ coefficients in linear regression be used to estimate the $R^2$?

I am trying to interpret the results of an article, where they applied multiple regression to predict various outcomes. However the $\beta$'s (standardized B coefficients defined as $\beta_{x_1} = B_{...

1
2 3 4 5
18