3
$\begingroup$

We are in a regression setting. Let's start by defining some notation and terminology.

$y_i$ is observation $i$ of some (response) variable $Y$.

$\hat{y}_i$ is the value of $y_i$ predicted by a regression.

$\bar{y}$ is the average of all observations of $Y$.

$$ y_i-\bar{y} = (y_i - \hat{y_i} + \hat{y_i} - \bar{y}) = (y_i - \hat{y_i}) + (\hat{y_i} - \bar{y}) $$

$$( y_i-\bar{y})^2 = \Big[ (y_i - \hat{y_i}) + (\hat{y_i} - \bar{y}) \Big]^2 = (y_i - \hat{y_i})^2 + (\hat{y_i} - \bar{y})^2 + 2(y_i - \hat{y_i})(\hat{y_i} - \bar{y}) $$

$$SSTotal := \sum_i ( y_i-\bar{y})^2 = \sum_i(y_i - \hat{y_i})^2 + \sum_i(\hat{y_i} - \bar{y})^2 + 2\sum_i\Big[ (y_i - \hat{y_i})(\hat{y_i} - \bar{y}) \Big]$$

$$ SSRes := \sum_i(y_i - \hat{y_i})^2 $$$$ SSReg := \sum_i(\hat{y_i} - \bar{y})^2 $$$$ Other = 2\sum_i\Big[ (y_i - \hat{y_i})(\hat{y_i} - \bar{y}) \Big] $$

The interpretation of the $SSRes$ seems straightforward enough, just the sum of the squared differences between the predicted and the true values. Why we would square these instead of taking the absolute value is not immediately obvious, but it at least makes sense why we would care about the difference between the true and predicted values.

What intuition is there for $SSReg?$ Why should we care about the distance between the predicted values and the average value? Further, what does this have to do with an "explained" sum of squares? What is being explained?

I can wrap my head around this when $Other = 0$, such as in OLS linear regression, since the $SSReg$ and $SSRes$ add up to the total sum of squares. When $Other \ne 0$, however, I am unsure how to interpret the $SSTotal$ decomposition beyond the $SSRes$.

$\endgroup$
3
  • 3
    $\begingroup$ Re "Why should we care about the distance between the predicted values and the average value?" Because that's the simplest possible OLS model (based on the data, anyway), and thereby serves as a unique frame of reference. When the mean predicts the data adequately, you aren't obliged to complicate your analysis by throwing in more variables. $\endgroup$
    – whuber
    Jul 5, 2023 at 14:11
  • $\begingroup$ If you turn classic books of yore, they always mentioned it as the reduction in sum of squares. That is the amount of sum of squares that could reduced from the total sum of squares "attributable to having fitted the regression". $\endgroup$ Jul 5, 2023 at 14:39
  • $\begingroup$ @User1865345-solidarityMods I buy that for OLS linear regression, but what about when $Other \ne 0?$ $\endgroup$
    – Dave
    Jul 5, 2023 at 16:56

1 Answer 1

0
$\begingroup$

"Explained" in the sense that SS_regression reflects/measures variability in the dependent variable Y that is "accounted for" by the independent variables (predictors, X-variables). In linear regression, the Y-hat variable represents the linear combination of the independent (X) variables (and potentially their interactions and/or non-linear terms).

$\endgroup$
3
  • $\begingroup$ In what way does $SSRes$ consider what has been accounted for? Just because it’s not the residual $SSRes?$ But then what about that “other” term in the decomposition of the total sum of squares? $\endgroup$
    – Dave
    Jul 5, 2023 at 16:48
  • $\begingroup$ I'm not sure when there would be an "other" term that isn't zero. By definition in OLS regression, residuals (e = y - y-hat) and y-hat scores are uncorrelated. So there would be no covariation that could be reflected by your "other" term. Therefore, SS_res represents all remaining variability in Y after accounting for the independent variables whose effects are "summarized" in the Y-hat variable. Another way to understand this is that the multiple correlation R in OLS regression is equal to the bivariate correlation between Y and Y-hat. $\endgroup$ Jul 5, 2023 at 19:36
  • $\begingroup$ $Other=0$ is the exception, not the rule. You will get $Other\ne 0$ in ridge or LASSO regression, for instance. $\endgroup$
    – Dave
    Jul 5, 2023 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.