7
$\begingroup$

Let's say I perform linear regression on some data that produces the following $R^2$:

$\text{RSS} = 1966815.13$

$\text{TSS} = 2145213.91$

$R^2 = 0.083$

Now let's say I bucket (take the average of successive equal size windows) the data and perform the exact same linear regression. This time the $R^2$ is:

$\text{RSS} = 1187.56$

$\text{TSS} = 4758.32$

$R^2 = 0.75$

What have I just done here?

I clearly haven't improved the explanatory power of the model over the original data, yet I've hacked the $R^2$ by averaging. What's the bias / variance argument of why this occurs?

A graphical representation:

enter image description here

Edit: I've reduced the total variance of the data, but the bias remains unaffected. When (if ever) is this a valid statistical approach?

$\endgroup$
6
  • $\begingroup$ What do you mean by “bucket” the data. Clearly you’ve taken some kind of average; what exactly did you do? $\endgroup$ – Dave Oct 9 '20 at 12:21
  • 10
    $\begingroup$ You've changed your data, by averaging it, your model now is better at explaining/predicting the average, which makes sense, since the average always has lower variability than the original data. $\endgroup$ – user2974951 Oct 9 '20 at 12:22
  • 3
    $\begingroup$ This is one of the many reasons why you should never trust only R2. $\endgroup$ – osmoc Oct 9 '20 at 12:50
  • 7
    $\begingroup$ This is related to the Ecological Fallacy. $\endgroup$ – whuber Oct 9 '20 at 14:04
  • 1
    $\begingroup$ Not exactly the same question, but reminds me of statmodeling.stat.columbia.edu/2020/09/26/… $\endgroup$ – Adrian Oct 10 '20 at 7:16
8
$\begingroup$

You decreased the (total) variance in the data making it easier to explain.

Thanks to the same $y$-axis limits, it can be easily seen that the data of your first example is much more spread in this direction than in the $x$-axis direction. Your linear regression model does capture the slight trend of increasing $y$ with increasing $x$, but it doesn't tell you anything about the variation along the $y$-axis when differences in $x$ are small, i.e. around the regression line. This is measured by residual sum of squares ($\text{RSS}$) you included.

In other words, there is still a lot of "error", which hasn't been accounted for in this model. Since $R^2$ represents the ratio between the explained variance and total variance, it remains small.

In your second set of data, most of the variation is explained by that same linear relationship between $x$ and $y$. Only a small part of the variance is left unexplained. This is reflected both in a smaller ratio $\frac{\text{RSS}}{\text{TSS}}$ as well as a smaller value of total sum of squares ($\text{TSS}$) as well.

To conclude, the same model performs much better in the second case, because that set of data is much easier to explain (with this type of model).


You also asked when, if ever, this would be a good statistical approach. It depends on what problem you are approaching with this.

If you wanted to show that your model fits well to your data and only selected a subset of data for which it performs particularly well, that is an example of cherry picking, which is a deceitful practice when done intentionally.

On the other hand, if you consider the variation in the direction of $y$-axis to be noise and you just want to give a succinct summary of your data, it might be acceptable to give it in some sort of average (like binning your data as you did). However, the regression line also serves the purpose of illustrating the upward trend well without manipulating the data itself. It also makes clear what is your data and what your model: the assumption of $y$ variance being noise (or error) is implicit here.

$\endgroup$
1
$\begingroup$

The edited question asks: "When (if ever) is this a valid statistical approach?"

I can't think of a good reason.

One COULD argue that, by taking averages by binning, you've "de-noised" the data. However, that implicitly assumes that all of the observed variance is noise. That seems like a highly dubious assumption.

By eye, your alt-model shows the same linear trend that the model on the raw data does. So you haven't gained any understandings there. But your 2nd model greatly overstates its explanatory power, and I can't envision a use case where that is ever a legit thing to do.

Also ... binning/averaging the data like this before the modeling process essentially eliminates the possibility that you can find OTHER predictor variables that will improve your model.

If the best model you can fit has an R-sq of 0.083 ... so be it. It happens. Some data series just inherently have a lot of unexplainable noise.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.