Why a small RSS indicates good linearity in general linear regression?

Question

Consider a generalized linear regression problem where $EY=Z\beta$.

The Residual Sum of Squares (RSS) is $\text{RSS}:=\xi^T\xi$, where $\xi:=Y-\hat{Y}=\{I-Z(Z^TZ)^{-1}Z^T\}Y$ is the regression residual.

My first question is:

• Why a small RSS indicates good linearity between $Y$ and $Z$?

Besides, we have:

$$E\xi=\{I-Z(Z^TZ)^{-1}Z^T\}EY$$

As a result, as long as $EY$ can be written as $EY=Zf(Z)$, $E\xi$ is always zero. This means the expectation of the residual is always zero no matter linearity or non-linearity. Is this right?

Please explain by formulars.

Answer 1

It doesn't!

library(ggplot2)
N <- 1000
z <- runif(N, 0, 50)
y <- 0.02*z^2 
L <- lm(y ~ z)
d <- data.frame(
  Z = z,
  Y = y
)
ggplot(d, aes(x = Z, y = Y)) +
  geom_point() +
  geom_abline(
    intercept = summary(L)$coef[1, 1],
    slope = summary(L)$coef[2, 1]
    )
summary(L)$r.squared # 0.9383307

In the above example, I get $R^2>0.93$, and $R^2$ is a transformation of the residual sum of squares. While it is hard to say exactly what constitutes a good $R^2$ value or a good (small) residual sum of squares, $R^2>0.93$ is usually considered quite high, indicating a small residual sum of squares. Thus, there must be a good linear relationship between the prediction $Z$ and outcome $Y$, right? Wrong!

The points quite clearly do not fit a line, which is consistent with the simulation giving the relationship as being quadratic.

What a small residual sum of squares does tell you is that the predictions are generally close to the observed values. It might be that you could do better, such as fitting a quadratic term to the above regression.

This great post gives another example and further elaboration. While the question there explicitly deals with $R^2$ and not the residual sum of squares, they are functions of each other.

$$ \text{Residual Sum of Squares} =\overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2\\ \text{Total Sum of Squares} = \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 $$$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right)=1-\left(\dfrac{ \text{Residual Sum of Squares} }{ \text{Total Sum of Squares} }\right)\\ \Big\Updownarrow\\ \left[\text{Residual Sum of Squares}\right] = (R^2 - 1)\left[\text{Total Sum of Squares}\right] $$