# Simple Regression: how to prove that adding an observation that exactly follows the regression line never decreases the magnitude of the correlation?

Suppose we fit by least square a regression line to $$n$$ pairs of $$(x_i,y_i)$$ observations, with

$$\hat{y}_i = \hat{\beta}_0 + x_i \hat{\beta}_1$$

Now suppose we add a single observation $$(x_{n+1}, y_{n+1})$$ that fits the regression line perfectly,

$$y_{n+1} - \hat{y}_{n+1} = 0$$

where $$\hat{y}_{n+1}$$ is calculated from the regression coefficients obtained from the previous $$n$$ observations only.

How can I prove that adding this new observation will never decrease the magnitude of the correlation $$r$$ between $$x$$ and $$y$$?

The magnitude of the correlation cannot decrease by adding a value that perfectly fits the regression line.

First, we note that the extra observation can't change the sign of the simple regression slope. This is because the slope $$\hat{\beta}_1$$ won't be affected by the new observation, and the sample correlation $$r$$ equals $$\hat{\beta}_1$$ times a positive constant. Without loss of generality, we assume $$r \ge 0$$.

Next, consider the coefficient of determination $$R^2$$. In simple linear regression, the coefficient of determination $$R^2$$ is the square of the correlation $$r$$ (see this proof for details).

Because the square is monotonous function over the $$[0, 1]$$ interval, proving that the magnitude of the correlation cannot decrease is equivalent to proving that $$R^2$$ cannot decrease.

Denote as $$R^2_n$$ the coefficient of determination, and $$\bar{Y}_n$$ the mean of $$Y$$, computed from the first $$n$$ data points. We can write

\begin{align} R^2_n &= 1 - \frac{\sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2 }{\sum^{n}_{i=1} ({Y}_i - \bar{Y}_n)^2} \\ &= 1 - \frac{\sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2 }{\sum^{n}_{i=1} (\hat{Y}_i - \bar{Y}_n)^2 + \sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2} \end{align}

Now, consider the coefficient of determination $$R^2_{n+1}$$ and mean $$\bar{Y}_{n+1}$$ based on the $$n+1$$ observations. We can write

\begin{align} R^2_{n+1} &= 1 - \frac{\sum^{n+1}_{i=1} ({Y}_i - \hat{Y}_i)^2 }{\sum^{n+1}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 + \sum^{n+1}_{i=1} ({Y}_i - \hat{Y}_i)^2} \\ &= 1 - \frac{\sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2 }{\sum^{n+1}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 + \sum^{n}_{i=1} ({Y}_i - \hat{Y}_i)^2 } \end{align}

To prove that the magnitude of the correlation doesn't decrease, it is sufficient to prove that

$$\sum^{n+1}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 \ge \sum^{n}_{i=1} (\hat{Y}_i - \bar{Y}_n)^2$$

To do so, we see that that

$$\sum^{n+1}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 \ge \sum^{n}_{i=1} (\hat{Y}_i - \bar{Y}_{n+1})^2 \ge \sum^{n}_{i=1} (\hat{Y}_i - \bar{Y}_{n})^2$$

Where the last inequality is true because $$\bar{Y}_{n}$$ is the minimizer of $$\sum^{n}_{i=1} (\hat{Y}_i - k)^2$$.

This proves that the magnitude of the correlation can't decrease. We also note that the previous inequalities are strict when $$x_{n+1} \ne \bar{x}_n$$; we conclude that the correlation will not change if and only if either $$x_{n+1} = \bar{x}_n$$ or $$r = 0$$, and that the correlation will increase otherwise.

• You can simplify your argument by observing that the new parameter estimates must equal the old parameter estimates, whence the sum of squares of residuals remains the same. – whuber Jan 14 at 19:49