Timeline for How can you handle unstable $\beta$ estimates in linear regression with high multi-collinearity without throwing out variables?
Current License: CC BY-SA 3.0
22 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 19, 2012 at 14:35 | comment | added | Macro | @Luna, I don't particularly disagree with using ridge regression - this was just what first occurred to me (I answered before that was suggested). One thing I can say is that ridge regression estimate are biased, so, in some sense, you're actually estimating a slightly different (shrunken) quantity than you are with ordinary regression, making the interpretation of the coefficients perhaps more challenging (as gung alludes to). Also, what I've described here only requires understanding of basic linear regression and may be more intuitively appealing to some. | |
| Jul 19, 2012 at 14:21 | comment | added | Luna | Thanks Macro. Could you please shed some lights on why this approach might be better than Ridge Regression proposed above? Thank you! | |
| Jul 18, 2012 at 18:00 | comment | added | Macro | @Luna, yes, I guess what I learned is that you need to bear in mind what space you're referring to when you say that $x$ is orthogonal to something :) Glad I (and cardinal) could help. | |
| Jul 18, 2012 at 17:55 | comment | added | Luna | Hi Macro, Thank you for the excellent proof. Yeah now I understand it. When we talk about sample correlation between x and residuals, it requires the intercept term to be included for the sample correlation to be 0. On the other hand, when we talk about orthogonality between x and residuals, it doesn't require the intercept term to be included, for the orthogonality to hold. | |
| Jul 18, 2012 at 17:48 | comment | added | Macro | @cardinal, yes of course, I clumsily (i.e. inprecisely to the point of possibly being wrong) was trying to say that in a chat with Luna. While making this derivation, the thing that threw me was that $\sum x_i e_i = 0$ in the model with no intercept, meaning the residuals are orthogonal to the space defined by $bx$ (not the space defined by $a+bx$). But, in that case, it's not equivalent to the correlation being zero - it took me a minute to see that. | |
| Jul 18, 2012 at 17:44 | comment | added | cardinal | @Macro: Regarding your edit, geometrically it's "easy" to see. The residuals are perpendicular to $(\mathbf 1, \mathbf x)$ when an intercept is included and $\mathbf x - \bar x \mathbf 1$ lies in the subspace generated by the aforementioned pair. If there is no intercept in the model then the residuals are no longer perpendicular to this subspace---the vector has been tilted relative to the subspace and so there is no longer a right angle between them. | |
| Jul 18, 2012 at 17:19 | history | edited | Macro | CC BY-SA 3.0 |
added 1 characters in body
|
| Jul 18, 2012 at 17:13 | comment | added | Macro | @Luna, please see my edit. | |
| Jul 18, 2012 at 17:13 | history | edited | Macro | CC BY-SA 3.0 |
added 1564 characters in body; added 64 characters in body; added 74 characters in body
|
| Jul 18, 2012 at 15:47 | comment | added | Luna | Hi Macro, if you throw in an intercept term, then the residuals are orthogonal to the space spanned by X and the 1's. If there is no intercept term, then the residuals are orthogonal to the space spanned by X alone. So my question is: why do you want to include the intercept term? Thank you! | |
| Jul 17, 2012 at 22:59 | comment | added | Macro | @Luna, I guess my response is - why wouldn't there be a non-zero correlation? In the case of regression with the intercept, the residuals are exactly orthogonal to the space spanned by $X$, therefore the sample correlation between the residuals and any predictor will be exactly 0 - see this figure: en.wikipedia.org/wiki/File:OLS_geometric_interpretation.svg but when you remove the intercept that is no longer true. | |
| Jul 17, 2012 at 22:28 | comment | added | Luna | Why will there be a non-zero correlation? Thank you! | |
| Jul 17, 2012 at 20:18 | comment | added | Macro | @Luna, no, I don't think so. If you exclude $\alpha_0$ from the model then $\eta_i$ and $X_i$ will have a non-zero correlation. The logic underlying my answer will only apply if you keep the intercept. | |
| Jul 17, 2012 at 20:16 | comment | added | Luna | @Macro: if you don't include the intercept alpha_0 there, you can get the same results, right? | |
| Jul 17, 2012 at 18:33 | comment | added | Macro | @Luna, that is to guarantee that $\eta_i$ is uncorrelated with $X_i$ and so that $\eta_i$ can be interpreted as "the part of $Z_i$ not explained by linear relationship with $X_i$" | |
| Jul 17, 2012 at 18:32 | comment | added | Luna | Thank you Macro. In your regression of Zi fitting onto Xi, why do you include the intercept? | |
| Jul 17, 2012 at 16:24 | comment | added | Macro | @whuber, I see your point now but all else held equal I guess I'd prefer this, particularly since the problem description restricted us to the convenient (but not implausible) world where there are only two collinear predictors :) My reason is that I've always found the inability to intuitively interpret PCs as predictors to be a drawback. In this case, you at least can interpret $\hat \theta_1$ in the usual way and can, with some qualification, interpret $\hat \theta_2$ as well. | |
| Jul 17, 2012 at 16:22 | comment | added | whuber♦ | One thing I had in mind is that PCA generalizes easily to more than two variables. Another is that it treats $X$ and $Z$ symmetrically, whereas your proposal appears arbitrarily to single out one of these variables. Another thought was that PCA provides a disciplined way to reduce the number of variables (although one must be cautious about that, because a small principal component may be highly correlated with the dependent variable). | |
| Jul 17, 2012 at 16:18 | comment | added | Macro | @whuber, yes but I think this is simpler since all you need to know is linear regression for this to make sense and you still (like the PCs) have the nice property that $\hat \theta_1$ and $\hat \theta_2$ are not correlated, unlike if you'd entered $X_i,Z_i$ into the model directly. Perhaps I'm missing your point. | |
| Jul 17, 2012 at 16:15 | comment | added | whuber♦ | This sounds like an approximation to replacing $(X, Z)$ by their principal components. | |
| Jul 17, 2012 at 15:58 | comment | added | Andy W | This reminds me of partial regression plots. | |
| Jul 17, 2012 at 15:46 | history | answered | Macro | CC BY-SA 3.0 |
