Questions tagged [multicollinearity]

Situation when there is strong linear relationship among predictor variables, so that their correlation matrix becomes (almost) singular. This "ill condition" makes it hard to determine the unique role each of the predictors is playing: estimation problems arise and standard errors are increased. Bivariately very high correlated predictors are one example of multicollinearity.

Filter by
Sorted by
Tagged with
0 votes
1 answer
29 views

How does perfect multicollinearity affect $R^2$ and $R_{\text{adj}}^2$?

I'd like to know how does perfect collinearity affect measures of fit (R squared and R squared adjusted). A mathematical approach is not necessary, just the general intuition is fine.
user avatar
1 vote
0 answers
25 views

Is there a nonlinear counterpart of multicollinearity? [closed]

Is there a nonlinear counterpart of multicolinearity. i.e. if two variables x1 and x2 are nonlinearly dependent on eachother in such a way that makes interpretation the coefficients of a specific ...
user avatar
0 votes
0 answers
9 views

Multicollinearity and factor analysis

I’m carrying out and principal axis factor analysis to validate an existing questionnaire in a different population. The determinant of the R-matrix is incredibly low and way below the threshold 0....
user avatar
  • 1
1 vote
2 answers
38 views

How to measure marginal effect of interdependent variables on a binary outcome?

My data represents observations on a possible sequences of events that may lead to a positive outcome (y). Each event (A, B, C, D) is dependent on the previous event; for D to occur C must occur, for ...
user avatar
0 votes
0 answers
27 views

Multicollinearity in polynomial regression in R

I fit a model to my data with the following formula after a stepwise selection (x1 to x5 stand for my variables): lm(formula = outcome ~ x1+ x3 + x4 + x2+ x5+ poly(x4, 2) + poly(x2, 2) + poly(x5, 2) + ...
user avatar
  • 1
0 votes
0 answers
19 views

GMM clustering with binary and multicollinear data

I am using GMM clustering on bank data. The data have both categorical and numerical attributes. The categorical data were converted to numerical using binary encoding. I have a couple of questions: ...
user avatar
0 votes
0 answers
23 views

interaction between (possible) correlated variables

Suppose I have a set of variables a,b,c,d,f,g,h,i, and I created a composite score by summing these eight variables into a composite score e=(a+b+c+d+f+g+h+i). Now I want to explore the interaction on ...
user avatar
4 votes
0 answers
48 views

PCA versus mixed effects model: Incorporating relationship between loadings?

I have species abundances with associations between environmental variables. I realize the RDA will only be able to tell you the strength of the relationship of all the species abundances with the ...
user avatar
1 vote
1 answer
49 views

Multicolinearity ONLY raises the variance of the coefficient estimates?

A hypothetical conversation: Person A: I am building a forecasting model. It is a logistic regression. All coefficients are statistically significant at the 5% level. I calculated the VIF for the set ...
user avatar
0 votes
0 answers
14 views

Assessing combined effects of binary predictor variables

I'm looking to analyse the effect of 20-30 binary predictor variables on a continuous response variable. I'm no statistician, so first off I'm not sure whether a regression with this many binary ...
user avatar
0 votes
0 answers
29 views

How to adjust regression for colinearity within certain levels of categorical variable?

I'm analyzing results from a large workforce survey, with approximately 110,000 responses to each question. We are interested in how certain demographic variables impact employees' opinions, so we ...
user avatar
0 votes
0 answers
39 views

Interpreting Logistic Regression Coefficients Under Collinearity

I thought this would be an easy question to find an answer to, but for the life of me I am having trouble finding anything that fully addresses my current problem: Consider a situation where we are ...
user avatar
0 votes
0 answers
19 views

Multicollinearity: not by some measures (VIF, TOL), yes by others (Wi, Fi) - does this invalidate the regression model?

Data is from http://peopleanalytics-regression-book.org/data/sociological_data.csv After reading this into R as sociological_data, here is my code: ...
user avatar
0 votes
1 answer
53 views

Multicollinearity iff determinant correlation matrix = 0

I'm studying Linear Models again, after finishing my degree some years ago. I found in my old notes that, according to my professor, one can check multicollinearity calculating the determinant of the ...
user avatar
0 votes
1 answer
61 views

Why do I find that OLS linear regression is robust against colinearity?

As per the textbook, OLS should fail when using colinear covariates. On their LinearRegression() documentation, sklearn states: When features are correlated and the columns of the design matrix have ...
user avatar
  • 105
1 vote
0 answers
46 views

Is the non-multicollinearity assumption for OLS multiple regression just an assumption of convenience?

The four assumptions for bivariate regression are:     • (L)inearity     • (I)ndepdent observations     • (N)ormal errors     • (E)qual variance And for multiple regression we add a fifth assumption:  ...
user avatar
  • 3,863
0 votes
0 answers
15 views

Coefficients interpretation after recovering raw coefficients from a regression which used orthogonal regressors

I have an unbalanced panel of 747 observations and 15 years. After testing for Pooled, FE and RE, FE is the "best" model. However, I have multicollinearity problems. I can either remove one ...
user avatar
0 votes
1 answer
29 views

If a regression problem is ill-conditioned, does that mean we cannot perform SGD? What happens if we do?

By ill-conditioned regression problem, I mean that the feature matrix $X$ is not full rank. For example, X contains two or more columns highly correlated. If that's the case, $X^T\cdot X$ is not ...
user avatar
  • 143
1 vote
0 answers
14 views

When do control variables increase precision?

Suppose we're interested in the effect $\beta$ of a treatment $D$. To increase the precision of our estimate (ie., reduce the variance of $\hat{\beta}$), we can include a control variable $X$ that ...
user avatar
1 vote
0 answers
30 views

Multicollinearity with fixed effects

I have a dataset with N=90,000 observations, where each observation is a respondent in a survey. Respondents vary across multiple countries (50) and years (8). Each respondent was only surveyed once, ...
user avatar
  • 11
0 votes
0 answers
19 views

Should I remove Collinear variables for Random Forests

I am aware that Random Forests aren’t typically affected by collinearity issues, but I am trying to reduce how many variables I am utilising in my RF model. There are variables that are obviously ...
user avatar
0 votes
0 answers
24 views

"Residualize out" vs. "regress out" vs. "covary out"

I have heard the term "residualize out". I believe residualizing X out from Y means to regress Y ~ X and use the residuals in place of the original Y. How does this compare to "...
user avatar
  • 21
0 votes
1 answer
95 views

How to interpret concurvity and wiggliness induced by low or high K knots in GAM models

Two questions: 1. Concurvity for factors =1, is this normal? 2. How do you interpret a partial effect when the effect is linear and lines for confidence intervals are pinched at 0? Is it okay for ...
user avatar
0 votes
1 answer
74 views

Proof that multicollinearity doesn't produce biased estimators

I'm trying to prove that multicollinearity doesn't introduce bias into a multiple linear regression model, but my proof seems to indicate the opposite. If we represent the model as $$y = \hat \beta_0 +...
user avatar
0 votes
0 answers
23 views

Does the condition number depend on the sample size?

I am working with a dataset of 214 thousand firms and I am running a logistic regression. Not only my dependent variable (bOptingOut), but also some of my independent variables are binary (...
user avatar
1 vote
1 answer
24 views

Does multicollinearity affect EDA?

I have been working on a dataset pertaining to 'churn analysis'. I have been trying to demonstrate whether the customers that are being charged more are also the ones that churn more or not. My ...
user avatar
1 vote
1 answer
105 views

MLR - Eliminating multicollinearity when predictors are transformations of others

I am applying a multiple linear regression on a data set, where some of the predictors are "transformations" of others (however, I'm not entirely sure if they are linear transformations or ...
user avatar
2 votes
2 answers
169 views

PCA can make VIF worse? Why is it so often sold as a multicollinearity fix?

It's the conventional wisdom that a PCA transformation can cure multicollinearity. Putting this into practice on example data, I find myself confused. In the following case, applying PCA seems to have ...
user avatar
0 votes
1 answer
115 views

Is there a difference between perfect collinearity and multicollinearity?

I've read that for multiple regression analysis there is an assumption of no perfect collinearity. Is that the same as multicollinearity?
user avatar
0 votes
0 answers
41 views

VIFs in Fixed-Effects-model

I know that R is providing some Warnings messages when you try to get your VIFs to test for multicollinearity. But why is that? Does someone know a good paper or the answer itself? CODE: ...
user avatar
0 votes
0 answers
17 views

Collinearity and relationship between the parameter estimates [duplicate]

Given a dataset with a dependent variable (Y), and three independent variables/features {X1,X2,X3}, where collinearity exists, and if we fit it using linear regression, is there any relationship ...
user avatar
0 votes
0 answers
42 views

Logistic regression - are interaction terms redundant vs original features if using L1 penalization for feature selection?

I am running lasso/elastic regression for feature selection in a logistic classifier. I have two continuous features, and was wondering if it would be redundant to include an interaction term or other ...
user avatar
0 votes
0 answers
27 views

Regress out covariates in a different order

I got a list of covariates in the multiple linear regression Y = X + cov1 + cov2 + ... + cov10 One of the covariates (e.g. cov1) is highly correlated with Y (Pearson's correlation = ~0.6). I was ...
user avatar
2 votes
1 answer
822 views

Is this the correct way to estimate a Difference-in-Differences model in R (multiple periods and states)?

First, I would like to say that I'm not sure if this is a question for CrossValidated or StackOverflow, as the question relates both to coding and statistics. My Q: I would like to know if this is the ...
user avatar
  • 145
1 vote
1 answer
93 views

Do you have to remove perfectly collinear independent variables prior to Cox regression?

Suppose you have independent variables that change only with each new time step (and possibly others that change freely): ...
user avatar
0 votes
0 answers
63 views

Which threshold should I use for GVIF1/(2⋅df)? (Variance Inflation Factor)

I'm using the mtcars dataset in R, I used the car packages to estimate the VIF, but since I have factor variables I got the vif table with GVIF and GVIF1/(2⋅df) values, in another question Which ...
user avatar
  • 1
21 votes
6 answers
2k views

Why is dimensionality reduction used if it almost always reduces the explained variation?

Let's say I have $N$ covariates in my regression model, and they explain 95% of the variation of the target set, i.e. $r^2=0.95$. If there are multicollinearity between these covariates and PCA is ...
user avatar
2 votes
2 answers
197 views

When adding polynomial features, the issue of multicollinearity doesn't hold. Why?

In the regression model, sometimes to capture the non-linear relationship between dependent and independent variables, we use polynomial features. But in regression, if the two or more features are ...
user avatar
2 votes
1 answer
67 views

What happens to Lasso Regression when variables are collinear? How do we deal with it?

I'm self-studying the Elements of Statistical Learning and I came across this question I believe that rather than a single solution, there's now a manifold of solutions with $\hat{\beta}_j + \hat{\...
user avatar
1 vote
0 answers
13 views

Little to no collinearity (low VIF), but extremely high path coefficients

Consider a simple mediation model with three variables: independent variable, mediator, and dependent or outcome variable. Three paths between these variables: path a goes from independent variable to ...
user avatar
  • 11
0 votes
0 answers
16 views

Multicollinearity in a demand system

I have a question regarding a paper I am currently reading, which can be found here: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3819959. The authors mainly rely on cross-sectional price data ...
user avatar
  • 1
0 votes
0 answers
59 views

Is multicollinearity a problem when fitting a Bayesian regression model using ADVI?

If I’m fitting a bayesian regression model using ADVI, is it important to ensure all the covariates are uncorrelated with each other? I have a vague understanding that ADVI doesn’t play well with ...
user avatar
  • 29
2 votes
1 answer
54 views

Correlated regressors but coefficient estimation is still good, why?

I have read time and again that strong collinearity between regressors in OLS regression can result in inaccurate estimates for individual coefficients. To see this action, I wrote the following code ...
user avatar
  • 121
9 votes
4 answers
2k views

A linear pattern occurs on my residual plot: what can I do?

I'm a bit stuck with a problem here and any kind of help would help a lot :) Just to give a clue about my data. I have 6 independant variables (IV) which are: $X_1$ = Population -within a block- $X_2$...
user avatar
  • 91
1 vote
1 answer
29 views

Is there any way to determine VIF of some variable that is included in the dataset that has so many variables?

I am new in statistics and need some help to determine the VIF value on all my variables/features in the dataset with a lot of variables. I have 98 variables with 76 observations and need to find VIF ...
user avatar
0 votes
0 answers
53 views

Polynomial regression and multicollinearity [duplicate]

to use linear regression we should have the predictors/features be independent from one another. However, in polynomial regression, x^2 is perfectly correlated to x ( knowing x value gives us the ...
user avatar
0 votes
0 answers
52 views

Multicollinearity for Logistic Regression and Neural Network

I am looking to fit Logistic Regression (LR) and Neural Networks (NN) models in order to predict if there will be avalanches during a day (0 or 1 dependant variable) based on meteorological variables (...
user avatar
  • 31
1 vote
0 answers
32 views

How to handle high multicollinearity for logistic regression and neural nets [closed]

I am looking to fit multiple models (Random Forest (RF), Classification Trees (CT), Logistic Regression (LR) and Neural Networks (NN)) in order to predict if there will be avalanches during a day (0 ...
user avatar
  • 31
2 votes
1 answer
243 views

Ridge Trace Plot - Interpretation

In my research, I aimed to perform a regression model with four predictors and one response variable. When I verified a high collinearity among the predictors, I was instructed to handle this problem ...
user avatar
  • 47
0 votes
0 answers
103 views

Multicollinearity and fixed effects

I am dealing with some collinearity issues with a fixed effect negative binomial model. I am currently working with fixest package in R. In few words, many of my regressors are collinear with the ...
user avatar

1
2 3 4 5
22