Let $Y \in \mathbb{R}^p$ be a vector of $n$ draws from a standard normal distribution $\mathcal{N}(0,1)$. Let $w_1, w_2 \in \mathbb{R}^p$ be $n$ independent draws from a normal distribution with $\mathcal{N}(0,\sigma^2)$.

Now, we define $X_1 = Y + w_1$ and $X_2 = Y + w_2$ and let $X$ be a matrix with the columns $X_1, X_2$. If $\sigma^2 = 0$ we have $X_1 = X_2$. In this situation the linear model is unidentified, because we cannot invert the gram matrix $X^{\top}X$ to get the least squares estimate:

$$ \beta = (X^{\top}X)^{-1} X^{\top}y$$

Another way of looking at this is to say that any convex combination of $X_1$ and $X_2$ will give the same prediction:

$$ \beta_0 + \alpha X_1 + (1 - \alpha) X_2 = \beta_0 + \gamma X_1 + (1 - \gamma) X_2 $$

for all $\alpha, \gamma \in [0,1]$, because $X_1 = X_2$.

Now if we pick a variance $\sigma^2 > 0$ we have $X_1 \neq X_2$ and the model is identified.

Via a small simulation I see that both parameters have approximately the same mass:

n <- 1000000
y <- rnorm(n)
x1 <- y + rnorm(n, 0, .1)
x2 <- y + rnorm(n, 0, .1)
X <- cbind(x1, x2)
solve(t(X)%*%X) %*% t(X)%*%y
x1 0.4980466
x2 0.4969557

I am looking for an intuitive explanation of why this is the case. If $X_1 = X_2$ any convex combination of $X_1$ and $X_2$ gives the same prediction. However, when adding a tiny little bit of noise to both variables, so the model is identified, the optimal solution seems to be to give both predictors the same mass. It seems to me there must be a good reason for that.

In general of course I am trying to get an intuition of how multicollinearity translates into parameter estimates (if i have p predictors and all share the same piece of variance - I am imagining Venn diagrams here - by which beta will this piece be 'absorbed'?). I understand this for instance for forward-selection, where $X_i$ can only explain variance in $Y$ that is not yet explained by any beta $X_j$, $j < i$. This seems not to happen in the above example, because otherwise the first entered $X_i$ would explain almost all the variance and hence the beta of the variable that was entered second would be much smaller.

My apologies if this is something incredibly obvious that is already asked & answered elsewhere.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.