Say I have some data that follows a general linear model: $$ Y = XB + E $$ for which: $Y \in \Re^{n \times m}$, $X \in \Re^{n \times p}$ and $B \in \Re^{p \times m}$

Further, let's assume (1) that we have $ E \sim \mathcal{N}(0,\Sigma)$, with $\Sigma \in \Re^{m \times m} $ and not diagonal. That is, the noise $E$ is correlated between our $m$ response variables.

Finally, let's assume (2) that the independent variables (columns in $X$) are also correlated, but not to the point where $X$ would be rank deficient.

We can now estimate $B$ and $E$ given $X$ and $Y$: $$ \hat{B} = (X^TX)^{-1}X^TY $$ $$ \hat{E} = Y-X\hat{B} $$ and let's define: $$ \Omega = cov(\hat{E}) = \frac{1}{n}\hat{E}\hat{E}^T $$

What I have found by simulating data according to the assumptions described above, is that the top $p$ eigenvectors of $\Omega$ (i.e. those with the largest eigenvalues) tend to be correlated (more than expected by chance) with the rows of $\hat{B}$ (but not with $B$). For this to hold true, it appears to be necessary that the data satisfies both of the two assumptions I stated earlier (correlated noise and correlated independent variables) - either one on its own does not produce this result. Edit: turns out there was a slight mistake in my simulation code, and actually the only necessary assumption seems to be that the noise is correlated.

In short, the covariance matrix of the residuals from a multivariate, multiple linear regression appears to factorize in such a way that the factors are predicted by the estimation errors in $\hat{B}$, iff (1) the independent variables are correlated and (2) the noise in the outcome variables is correlated.

Can anybody provide me with a formal explanation as to why this happens, or tell me what this phenomenon is called? I can't seem to find it described anywhere, but perhaps that is because the obvious keywords for it (correlation, residuals, regression, etc.) aren't particularly distinctive.

  • $\begingroup$ what is the correlation? how many observations you're simulating? $\endgroup$ – Aksakal Mar 28 '16 at 14:39
  • $\begingroup$ The noise correlations can have any structure or magnitude; the effect is always there. At the moment I'm simulating p = 8, n = 100 and m = 1000, so 8 independent variables, 100 observations and 1000 response variables. But it seems to generalize to any p < n. $\endgroup$ – Ruben van Bergen Mar 28 '16 at 14:59
  • $\begingroup$ One issue with your data is that you have way more parameters to estimate than the data you have. You are estimating (8000 + 1000^2)/2 parameters. You don't nearly enough data to estimate this many parameters. $\endgroup$ – Aksakal Mar 28 '16 at 15:05
  • $\begingroup$ Okay, but then how does that lead to this result? And why only with correlated noise, and not when the noise is IID, even though the number of parameters and datapoints is the same? Am I missing something? $\endgroup$ – Ruben van Bergen Mar 28 '16 at 15:13
  • $\begingroup$ Also, eigen vector of error covariance matrix has m rows, while the parameter estimates have p rows. How do you calculate the covariance? $\endgroup$ – Aksakal Mar 28 '16 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.