4
$\begingroup$

Question about the Total, Explained, and Residual Sum of Squares. I am in the simple linear regression model.

Could you help me clarify why the residual sum of squares (SSE where E stands for errors)

$$SSE = \sum_{i=1}^n (\hat{Y}_{i}-Y_{i})^{2} $$

is the amount in variance of Y which is not explained by the model and why the explained sum of squares (SSR where R stands for regression)

$$SSR = \sum_{i=1}^n (\hat{Y}_{i}-\bar{Y})^{2} $$

is the amount in variance of Y which is explained by the model, where $Y_{i}$ are the observations, $\bar{y}$ is the esperance and $\hat{Y}_{i}$ are the predicted values in the model.

Improve this question
$\endgroup$
4
  • $\begingroup$ These are estimated amounts, not genuine variances. $\endgroup$
    – Xi'an
    Dec 31, 2014 at 13:59
  • $\begingroup$ this was clear to me. how does comparing estimated values to the mean does account for model whereas comparing estimated values to the observations does account for the amount not explained by the model ? $\endgroup$
    – kiriloff
    Dec 31, 2014 at 14:13
  • 2
    $\begingroup$ Deviation from the mean is called total variation. Squared sum of total variation is $$TSS=\sum_{i=1}^{n}(Y_i-\bar{Y})^2.$$If you decompose $Y_i$ in two parts as $Y=\hat{Y}+\hat{\varepsilon}$, $\hat{Y}$ is the part explained by the model, and $\hat{\varepsilon}$ is the unexplained counterpart.Total variation is then,$$TSS=\sum_{i=1}^{n}(\hat{Y}+\hat{\varepsilon}-\bar{Y})^2$$Since $\hat{\varepsilon} = Y-\hat{Y}$, sum of the squared total variation can be decomposed into $$TSS=SSR+SSE.$$So, technically, SSR is the amount in the TSS explained by the model. $\endgroup$
    – Khashaa
    Dec 31, 2014 at 14:48
  • 1
    $\begingroup$ This is also a statistical version of Pythagoras' theorem. $\endgroup$
    – Xi'an
    Dec 31, 2014 at 14:58

1 Answer 1

Reset to default
2
$\begingroup$

If you take a normal regression model, $$Y_i|X_i\sim\mathcal{N}(X_i^\text{T}\beta,\sigma^2),$$ the density of the data $(Y_1,\ldots,Y_n)$ writes as follows: \begin{align*}&\exp\left\{ -\frac{1}{2\sigma^2}\sum_{i=1}^n (Y_i-X_i^\text{T}\beta)^2 \right\}\\ &\qquad=\exp\left\{-\frac{1}{2\sigma^2}\sum_{i=1}^n [(Y_i-X_i^\text{T}\hat{\beta})^2 +(X_i^\text{T}\hat{\beta}-X_i^\text{T}\beta)^2]\right\}\\ &\qquad=\exp\left\{-\frac{1}{2\sigma^2}[\text{SSR}+\text{SSE}]\right\}\\ &\qquad=\exp\left\{-\frac{1}{2\sigma^2}\text{SSE}\right\}\times\exp\left\{-\frac{1}{2\sigma^2}\text{SSR}\right\}\end{align*} and only the first term depends on the parameter $\beta$ and hence characterises the model fit, while the second term is about the residual variability of the $Y_i$'s around their best prediction or projection, $X_i^\text{T}\hat{\beta}$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.