3
$\begingroup$

Suppose we fit by least square a regression line to $n$ pairs of $(x_i,y_i)$ observations, with

$$\hat{y}_i = \hat{\beta}_0 + x_i \hat{\beta}_1$$

Now suppose we add a single observation $(x_{n+1}, y_{n+1})$ that fits the regression line perfectly,

$$ y_{n+1} - \hat{y}_{n+1} = 0 $$

where $\hat{y}_{n+1}$ is calculated from the regression coefficients obtained from the previous $n$ observations only.

How can I prove that adding this new observation will never decrease the magnitude of the correlation $r$ between $x$ and $y$?

$\endgroup$
3
$\begingroup$

The magnitude of the correlation cannot decrease by adding a value that perfectly fits the regression line.

First, we note that the extra observation can't change the sign of the simple regression slope. This is because the slope $\hat{\beta}_1$ won't be affected by the new observation, and the sample correlation $r$ equals $\hat{\beta}_1$ times a positive constant. Without loss of generality, we assume $r \ge 0$.

Next, consider the coefficient of determination $R^2$. In simple linear regression, the coefficient of determination $R^2$ is the square of the correlation $r$ (see this proof for details).

Because the square is monotonous function over the $[0, 1]$ interval, proving that the magnitude of the correlation cannot decrease is equivalent to proving that $R^2$ cannot decrease.

Denote as $R^2_n$ the coefficient of determination, and $\bar{Y}_n$ the mean of $Y$, computed from the first $n$ data points. We can write

\begin{align} R^2_n &= 1 - \frac{\sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2 }{\sum^{n}_{i=1} ({Y}_i - \bar{Y}_n)^2} \\ &= 1 - \frac{\sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2 }{\sum^{n}_{i=1} (\hat{Y}_i - \bar{Y}_n)^2 + \sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2} \end{align}

Now, consider the coefficient of determination $R^2_{n+1}$ and mean $\bar{Y}_{n+1}$ based on the $n+1$ observations. We can write

\begin{align} R^2_{n+1} &= 1 - \frac{\sum^{n+1}_{i=1} ({Y}_i - \hat{Y}_i)^2 }{\sum^{n+1}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 + \sum^{n+1}_{i=1} ({Y}_i - \hat{Y}_i)^2} \\ &= 1 - \frac{\sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2 }{\sum^{n+1}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 + \sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2 } \end{align}

To prove that the magnitude of the correlation doesn't decrease, it is sufficient to prove that

$$ \sum^{n+1}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 \ge \sum^{n}_{i=1} (\hat{Y}_i - \bar{Y}_n)^2$$

To do so, we see that that

$$ \sum^{n+1}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 \ge \sum^{n}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 \ge \sum^{n}_{i=1} (\hat{Y}_i - \bar{Y}_{n})^2 $$

Where the last inequality is true because $\bar{Y}_{n}$ is the minimizer of $\sum^{n}_{i=1} (\hat{Y}_i - k)^2$.

This proves that the magnitude of the correlation can't decrease. We also note that the previous inequalities are strict when $x_{n+1} \ne \bar{x}_n$; we conclude that the correlation will not change if and only if either $x_{n+1} = \bar{x}_n$ or $r = 0$, and that the correlation will increase otherwise.

$\endgroup$
  • 1
    $\begingroup$ You can simplify your argument by observing that the new parameter estimates must equal the old parameter estimates, whence the sum of squares of residuals remains the same. $\endgroup$ – whuber Jan 14 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.