3
$\begingroup$

Consider a simple linear regression $Y_i = \beta_0 + \beta_1x_i + \epsilon_i$ where each $\epsilon_i \sim N(0, \sigma^2)$. The solution to the linear regression problem is given by $\pmb{\beta} = (\pmb{X}^{T}\pmb{X})^{-1}\pmb{X}^{T}\pmb{Y}$ and the variance-covariance matrix is $\text{Var}(\pmb{\beta}) = \sigma^2 (\pmb{X}^{T}\pmb{X})^{-1}$.

Provided $x_i \geq 0$ for each index $i$, I want to show that $\rho(\hat{\beta_0}, \hat{\beta_1}) \leq 0,$ where $\rho(\cdot, \cdot)$ denotes the correlation.


I tried expanded the correlation definition, and I think enough to show covariance is negative. However, I'm having trouble doing so. I've used the definition of correlation, and I think I need to make use of the fact that $x_i \geq 0$, but I'm not able to get anywhere.

Can anyone please help me?

$\endgroup$
1
  • $\begingroup$ Could you show us how far you got? What expression did you develop for $(X^\prime X)^{-1},$ for instance? All you need to do is inspect the sign of its off-diagonal entry (and for that purpose you don't even need to find the inverse--computing the adjugate of $X^\prime X$ will suffice, because the determinant of $X^\prime X$ clearly is positive). $\endgroup$
    – whuber
    Sep 24 '20 at 18:43
3
$\begingroup$

I will set forth the argument doing as little algebra as possible, working from basic definitions.

$\mathbf X$ is the model matrix: its rows are numerical representations of the observations and its columns are the regressor variables recorded for each observation. (The vector $y$ separately records the values of the response variable.) Usually $X$ also includes a constant column used to model the constant, or "intercept," term.

In Ordinary Least Squares regression there is just one regressor variable, often denoted $\mathbf x = (x_1,x_2,\ldots, x_n).$ Thus, placing the constant column first in the model matrix,

$$\mathbf{X} = \pmatrix{1&x_1 \\ 1&x_2 \\ \vdots & \vdots \\ 1 & x_n}\text{ and }\mathbf y = \pmatrix{y_1 \\ y_2 \\ \vdots \\ y_n}.$$

We can proceed only by supposing the $x_i$ aren't all the same number (which is implicit in the question). According to your formula $\sigma^2 \left(\mathbf X^\prime \mathbf X\right)^{-1},$ the variance of the coefficient estimates $\hat\beta=(\hat\beta_0,\hat\beta_1)$ is a positive multiple ($\sigma^2$) of the matrix inverse of

$$\mathbf X^\prime \mathbf X = \pmatrix{1 & 1 & \ldots & 1 \\ x_1 & x_2 & \ldots & x_n}\pmatrix{1&x_1 \\ 1&x_2 \\ \vdots & \vdots \\ 1 & x_n} = \pmatrix{n & \sum_{i=1}^n x_i \\ \sum_{i=1}^n x_i & \sum_{i=1}^n x_i^2},$$

as you can compute using the rule of matrix multiplication. (If you track through the remainder of this analysis closely, you can confirm that only one of these calculations is actually needed: the upper right corner of this matrix product is $(1,1,\ldots,1)(x_1,x_2,\ldots,x_n)^\prime = \sum x_i.$)

The inverse of $\mathbf X^\prime \mathbf X$ is its adjugate divided by its determinant, where

$$\operatorname{ad} \pmatrix{a&b\\c&d} = \pmatrix{d&-b\\-c&a}$$

and

$$\det \pmatrix{a&b\\c&d} = ad-bc.$$

You can readily confirm this by multiplication:

$$\left[\frac{1}{\det \pmatrix{a&b\\c&d}} \operatorname{ad} \pmatrix{a&b\\c&d}\right]\, \pmatrix{a&b\\c&d} = \frac{1}{ad-bc}\pmatrix{ad-bc & 0 \\ 0 & ad-bc}=\pmatrix{1&0\\0&1}.$$

We need to be concerned about the sign of the determinant. Since for any 2-vector $\mathbf v$ it is the case that

$$\mathbf v^\prime \left(\mathbf X^\prime \mathbf X\right)\mathbf v = \mathbf w^\prime \mathbf w = ||\mathbf w ||^2 \ge 0,$$

where $\mathbf w = \mathbf {X v},$ it follows from this (and from the non-constancy of $\mathbf x$) that $\mathbf X^\prime \mathbf X$ is positive definite, whence its determinant cannot be negative. This is a nice argument because it requires no calculation at all.

(Alternatively, if you are a glutton for doing algebra, you may compute

$$\det \mathbf X^\prime \mathbf X = n\sum_{i=1}^n x_i^2 - \left(\sum_{i=1}^n x_i\right)^2 = n^2 \operatorname{Var}(\mathbf x) \gt 0$$

which shows the same thing.)

The off-diagonal terms of the inverse of $\mathbf X^\prime \mathbf X$ therefore are positive multiples of $-\sum_{i=1}^n x_i.$ When you assume all the $x_i$ are non-negative (and at least one is positive), this expression clearly is negative. Consequently, according to the formula you cite, the covariance (and hence the correlation) of $\hat\beta_0$ and $\hat\beta_1$ must be negative, QED.

In reviewing the argument, please notice that only one extremely easy calculation was involved: we had to sum the $x_i$ when computing $\mathbf X^\prime \mathbf X.$

Geometrically, when you wiggle a line through the scatterplot of points $(x_i,y_i)$ in an effort to approximate those points, tilting the line up increases the slope $\hat\beta_1$ with a consequent decrease in the intercept $\hat\beta_0,$ which (since the $x_i$ are nonnegative) lies to the left of all the points in the scatterplot. That's what it means to be negatively correlated.

Figure

$\endgroup$
5
  • $\begingroup$ Because if the determinant were negative, the factor $1/\det(X^\prime X)$ that multiplies the adjugate would change the signs of all its components! $\endgroup$
    – whuber
    Sep 24 '20 at 21:48
  • $\begingroup$ How'd you get that the determinant equals $n^2 \sigma^2_x$? I can't get the last equality $\endgroup$
    – user295785
    Sep 24 '20 at 22:24
  • $\begingroup$ It's straightforward algebra. (In fact, in many references this is the definition of the variance.) Try it by writing out the sums when $n=2.$ Note that the variance of a set of numbers is not an estimator: the divisions are all by $n,$ with no factors of $n-1$ involved. $\endgroup$
    – whuber
    Sep 24 '20 at 22:27
  • $\begingroup$ I still don't see it. For $n = 2$, I get $n(x_1^2 + x_2^2) - (x_1 + x_2)^2 = (n - 1)(x_1^2 + x_2^2) - 2x_1x_2$. The variance is $\sum_{i} x_i^2 - \left(\frac{1}{n} \sum_i x_i)\right)^2$ $\endgroup$
    – user295785
    Sep 24 '20 at 22:47
  • $\begingroup$ The variance is $(1/n)\sum x_i^2 - \left(1/n \sum x_i\right)^2.$ $\endgroup$
    – whuber
    Sep 25 '20 at 12:50
1
$\begingroup$

Let's say that $x=(1,2,3)$. Then the $X$ matrix is $$\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}$$ and $X^TX$ is \begin{align*} X^TX&=\begin{bmatrix}1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}= \begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix} \\ &= \begin{bmatrix} n & \sum_ix_i \\ \sum_ix_i & \sum_ix_i^2 \end{bmatrix} \end{align*} Finally, $(X^TX)^{-1}$ is \begin{align*} (X^TX)^{-1}&=\frac16\begin{bmatrix} 14 & -6 \\ -6 & 3 \end{bmatrix} \\ &= \frac{1}{n\sum_ix_i^2-(\sum_ix_i)^2}\begin{bmatrix}\sum_ix_i^2 & -\sum_ix_i \\ -\sum_ix_i & n \end{bmatrix} \end{align*} In general (see Seber & Lee, Linear Regression Analysis, John Wiley & Sons, 2003, Example 4.6), putting $X=[1_n,X_1]$: $$X^TX=\begin{bmatrix}n & n\bar{x}^T \\ n\bar{x} & X_1^TX_1 \end{bmatrix}, \quad (X^TX)^{-1}=\begin{bmatrix} \frac1n+\bar{x}^TV^{-1}\bar{x} & -\bar{x}^TV^{-1} \\ -V^{-1}\bar{x} & V^{-1} \end{bmatrix} $$ where $V=\tilde{X}^T\tilde{X}$ and $\tilde{X}$ has tipical element $\tilde{x}_{ij}=x_{ij}-\bar{x}_j$.

Is this enough?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy