Intuition behind regression sum of squares

Question

$SSR = \sum_{i = 1}^{n} (\hat{Y}_i - \bar{Y})^2$ is the sum of squares of the difference between the fitted value and the average response variable. In other words, it measures how far the regression line is from $\bar{Y}$. Higher $SSR$ leads to higher $R^2$, the coefficient of determination, which corresponds to how well the model fits our data. I'm having trouble wrapping my mind around why the farther away the regression line is from the average $Y$ means that the model is a better fit.

Answer 1

Just a bit of a misunderstanding with the definitions, I believe:

\begin{align} \text{SST}_{\text{otal}} &= \color{red}{\text{SSE}_{\text{xplained}}}+\color{blue}{\text{SSR}_{\text{esidual}}}\\ \end{align}

or, equivalently,

\begin{align} \sum(y_i-\bar y)^2 &=\color{red}{\sum(\hat y_i-\bar y)^2}+\color{blue}{\sum(y_i-\hat y_i)^2} \end{align}

and

$\large \text{R}^2 = 1 - \frac{\text{SSR}_{\text{esidual}}}{\text{SST}_{\text{otal}}}$

So if the model explained all the variation, $\text{SSR}_{\text{esidual}}=\sum(y_i-\hat y_i)^2=0$, and $\bf R^2=1.$

From Wikipedia:

Suppose $r = 0.7$ then $R^2 = 0.49$ and it implies that $49\%$ of the variability between the two variables have been accounted for and the remaining $51\%$ of the variability is still unaccounted for.

The sum of the squared distances between the mean ($\bar Y$) and the fitted values ($\hat Y$) (the SSExplained) is the part of the distance from the mean to the actual value ($ Y$) (TSS) that the model has been able to account for. The difference between these two calculations, is the unexplained part of the variation (the residuals). If you take TSS as a fixed value, the higher the SSExplained, the lower the SSResidual, and hence the closer to 1 R.Square will be.

---

Here is some intuition, at the risk of actually making clear waters murky. In OLS we minimize distances to the points in the data cloud in an overdetermined system, rendering a line that fulfills $\text{SST}>\text{SSE}$. The difference is the $\text{SSR}$ (residuals).

But let's imagine a data "cloud" of three points, all perfectly aligned. Now, let's play a game of actually doing the opposite of an OLS: we are going to increase the error by proposing a line different from the line that goes through all the points, using the mean as a fulcrum. Remember that the OLS goes through the mean values $({\bf \bar X, \bar Y})$, which is the blue point in the middle, through which we draw a horizontal line. In this case, opposite to the expected situation in OLS and just to illustrate the point, we can see how by moving the line from having zero $\text{SSR}$ (all the variance, $\text{SST}$ accounted by the model (the line), $\text{SSE}$) on the left "column" of the diagram, we introduce residual errors (in red, on the right part of the diagram):

Logically, by minimizing errors, and in the typical situation of an overdetermined system, the $\text{SST}> \text{SSE}$, and the difference will correspond to the $\text{SSR}$.

---

Here is a quick example with a widely available data set in R:

fit = lm(mpg ~ wt, mtcars)
summary(fit)$r.square
[1] 0.7528328
> sse = sum((fitted(fit) - mean(mtcars$mpg))^2)
> ssr = sum((fitted(fit) - mtcars$mpg)^2)
> 1 - (ssr/(sse + ssr))
[1] 0.7528328

Answer 2

why do we want a big difference between ŷ and ȳ?

maybe the graphs A, B, C, and D can be intuitively useful by visualizing the differences or distances between the 1. systolic blood pressure of each person from the mean systolic blood pressure(y-ȳ), 2. between the systolic blood pressure of each person from the regression line (y-ŷ), 3. and between the regression line and the mean systolic blood pressure (ŷ-ȳ).

the sum of squared differences of each sbp from the mean is the total sum of squares (tss) as shown in graph A.

if serum cholesterol is added or fitted as a predictor (x), a regression line can be placed on the graph. the sum of squared differences of each sbp value from the regression line is the regression sum of squares or explained sum of squares (rss or ess) as shown in graph B.

if the sum of squared differences of each sbp value from the regression line is smaller than the total sum of squares, then the regression line (serum cholesterol) has a better fit to the data than the mean sbp. the better the fit of the regression line the smaller the residual sum of squares( graph C).

if all the sbp fall perfectly on the regression line, then the residual sum of squares is zero and the regression sum of squares or explained sum of squares is equal to the total sum of squares (graph D). this means that all variation in sbp can be explained by variation in serum cholesterol.

to address the question: why do we want a big difference between ŷ and ȳ?

as the residual sum of squares approaches zero, the total sum of squares shrink until it is equal to the regression sum of squares when the y = ŷ. it this case, the mean of ŷ = ȳ.

Answer 3

This is the note I wrote for self studying purpose. I don't have much time to improve this due to lack of my English ability. But I guess this would be helpful. So I just paste this here. I will add some details later.

linear models We can come up with several linear models with error $\vec \epsilon$

$\vec y=\vec \epsilon$ (It isn't model technically. There is no $\beta$s but I'd consider this as a linear model for explanation)

$\vec y=\beta_0 \vec 1+ \vec \epsilon$ (0th model)

$\vec y=\beta_0 \vec 1+ \beta_1 \vec x_1 +\vec \epsilon$ (1st model)

$\vec y=\beta_0 \vec 1 +\beta_1 \vec x_1 +...+\beta_n \vec x_n + \vec \epsilon$ (nth model)

$m$th model least square fit minimizing error $\vec \epsilon ' \vec \epsilon$

$\hat y_{(m)}=X_{(m)}\hat \beta_{(m)}$ (vector symbols omitted.) $X_{(m)}=[\vec 1 \ \ \vec x_1 \ \ \vec x_2 \ \ ... \ \ \vec x_m ]$ $\hat \beta_{(m)}=(X_{(m)}'X_{(m)})^{-1}X_{(m)}' \vec y=(\hat \beta_0 \ \ \hat \beta_1 \ \ ...\ \ \hat \beta_m)'$

$SS_{residual}=\sum(\hat y^2_{i(m)}-y_i)^2$

$0$th model least square fit. $\hat y_{(0)}=\vec 1(\vec 1' \vec 1)^{-1} \vec 1' \vec y=\bar y \vec 1$

---

What does regression really mean? Let's consider this :$\sum y_i^2$.

If there's no model we there would be no regression so every $y_i$ can be treated as an error. (In other words, we can say the model is 0.) Then total error would be $\sum y_i^2$

Now let's adopt 0th model which is we don't consider any regressors ($x$s) The error of 0th model is $\sum(\hat y_{i(0)}-y_i)^2=\sum(\bar y-y_i)^2$. We can explain the error $\sum y_i^2-\sum(\bar y-y_i)^2=\sum \bar y^2$ and this is the regression of model 0th.

We can extend this in the same way to the nth model like below equation.

$$\sum y_i^2 = \sum \bar{y}^2_{(0)}+\sum(\bar{y}_{(0)}-\hat y_{i(1)})^2+\sum(\hat y_{i(1)}-\hat y_{i(2)})^2+...+\sum(\hat y_{i(n-1)}-\hat y_{i(n)})^2+\sum(\hat y_{i(n)}-y_i)^2$$ proof> First prove that $\sum (\hat y_{i(n-1)}-\hat y_{i(n)})(\hat y_{i(n)}-y_i) =0$

On the right hand, except the last term, is the regression of nth model.

Note this : $\sum(\hat y_{i(n-1)}-\hat y_{i(n)})^2=(X_{(n-1)}\hat \beta_{(n-1)}-X_{(n)}\hat \beta_{(n)})'(X_{(n-1)}\hat \beta_{(n-1)}-X_{(n)}\hat \beta_{(n)})$

$=\vec y'X_{(n)}(X_{(n)}'X_{(n)})^{-1}X_{(n)}' \vec y-\vec y'X_{(n-1)}(X_{(n-1)}'X_{(n-1)})^{-1}X_{(n-1)}' \vec y$

$=\hat \beta_{(n)}'X_{(n)}'\vec y-\hat \beta_{(n-1)}'X_{(n-1)}'\vec y$

Using this we can reduce those terms.
Let the regression of nth model $SS_R(\hat \beta_{(n)})=\hat \beta_{(n)}'X_{(n)}'\vec y$. This is the regression sum of squares due to $\hat \beta_{(n)}$

$$\sum y_i^2 = SS_R(\hat \beta_{(n)})+\sum(\hat y_{i(n)}-y_i)^2$$

Now substract 0th model's regression from each side of the equation.

$SS_{total}=\sum (y_i-\bar y)^2 = SS_R(\hat \beta_{(n)} )-SS_R(\hat \beta_{(0)})+\sum(\hat y_{i(n)}-y_i)^2$

This is the equation we usually consider during ANOVA method.

Now we can see that $SS_R( ( \hat \beta_1 \ \ ...\ \ \hat \beta_n)')=SS_R(\hat \beta_{(n)} )-SS_R(\hat \beta_{(0)})$, extra sum of squares due to $( \hat \beta_1 \ \ ...\ \ \hat \beta_n)'$ given $\beta_{(0)}=\hat \beta_0 \vec 1 =\bar y \vec 1$

So I guess regression sum of squares is how more we can explain the data than 0th model.

---

Model with no intercept Here we don't consider 0th model.

$\vec y=\beta_1 \vec x_1+ \vec \epsilon$

By minimizing $\vec \epsilon'\vec \epsilon$ we can get

$\sum y_i^2 =\sum(\hat y_{i(1)})^2+\sum(\hat y_{i(1)}-y_i)^2$

So in this case $SS_R=\sum(\hat y_{i(1)})^2$