Autocorrelation of coefficients for strongly autocorrelated inputs? In Chapter 5 of  "The Elements of Statistical Learning" ("Basis Expansion and Regularization", pg 150"), it is written that

Since the input signals have fairly strong positive autocorrelation, this results in negative autocorrelation in the coefficients.

Can someone explain the theory of why this is the case?
 A: Linear regression assumes that the true model is:
$$y= x^T \beta + \epsilon$$ where $x, \beta$ are $p \times 1$ vectors and $\epsilon \sim N(0, \sigma)$. Then, the conditional variance of the least squares estimate is: $$\textrm{Var}(\hat{\beta}\mid X)=\sigma^2(X^T X)^{-1}$$ where $X$ is an $N \times p$ matrix of our observed $x$'s. Our goal is to find the covariance matrix of $\hat{\beta}$. 
Using law of total variance, $$Var(\hat{\beta})=E[Var(\hat{\beta}\mid X)] + Var(E[\hat{\beta}\mid X])$$ We know $\hat{\beta}$ is an unbiased estimator (ie. $E[\hat{\beta}\mid X] = \beta$) so the second term is 0. Thus, $$Var(\hat{\beta})=\sigma^2E[(X^T X)^{-1}]$$
To simplify the calculation, assume $p=2$ and $N=1$. Then $$(X^TX)^{-1}=
C_1 \left[ {
\begin{array}{cc}
   x_2^2 & -x_2x_1\\
   -x_1x_2 & x_1^2\\
  \end{array} 
} \right]
$$ where $C$ is some constant. Thus, $$Var(\hat{\beta})=
C_2\left[ {
\begin{array}{cc}
   \sigma_2^2+\mu_2^2 & -(Cov(x_1,x_2)+\mu_1\mu_2)\\
   -(Cov(x_1,x_2)+\mu_1\mu_2) & \sigma_1^2+\mu_1^2\\
  \end{array} 
} \right]
$$
In particular, $Cov(\hat{\beta}_1, \hat{\beta}_2)=-(Cov(x_1,x_2)+\mu_1\mu_2)$. So a very strong positive correlation of $x_1,x_2$ leads to a negative correlation of the coefficients.
