3
$\begingroup$

I ran a ridge regression model on a set of data across 6 groups.

As you can see, the overall R-squared is low. Because groups A and B make up the most of the data, I would think the overall R-squared should be around 90%. Why is it 77.7%?

enter image description here

Group   Count   R-squared
Overall 3315    77.7
A       1064    95.05
B       988     94.53
C       595     83.67
D       405     71.82
E       154     79.18
F       109     60.87
$\endgroup$
  • 1
    $\begingroup$ You should think of this graphically. Even if you have several predictors, the main issue is shown with one response and one predictor where it's perfectly possible to have higher predictability within each group than when you combine groups. A trivial but instructive extreme case has two groups each of two points with in each group different values on the two variables. Then each group is guaranteed $R^2$ of 1 but the whole isn't guaranteed that at all. $\endgroup$ – Nick Cox Oct 23 '18 at 13:55
  • $\begingroup$ Imagine four points defining a rectangle with sides parallel to the axes and with group A the top left and bottom right points and group B the other two. Then within-group correlations are 1 and $-$1 but the overall model has $R^2$ of 0. $\endgroup$ – Nick Cox Oct 23 '18 at 13:55
  • $\begingroup$ (I don't think using ridge regression affects the issue.) $\endgroup$ – Nick Cox Oct 23 '18 at 13:57
  • $\begingroup$ @NickCox So would my interpretation be: the groups are very different from each other so the overall R-squared is low, but within each group, the predictability is high? $\endgroup$ – SAKURA Oct 23 '18 at 13:59
  • 2
    $\begingroup$ My examples are contrived. I can't see your data. Your groups don't have to be very different from each other for the effect you observed. 77.7% for $R^2$ is very high for researchers in some fields, so standards are relative. $\endgroup$ – Nick Cox Oct 23 '18 at 14:02
5
$\begingroup$

You should think of this graphically. Even if you have several predictors, the main issue is shown with one response and one predictor where it's perfectly possible to have higher predictability within each group than when you combine groups.

A trivial but instructive extreme case has two groups each of two points with, in each group, different values on the two variables. Then each group is guaranteed $R^2$ of 1 (because the points define a straight line) but the whole isn't guaranteed that at all.

Imagine four points defining a rectangle with sides parallel to the axes and with group A the top right and bottom left points and group B the other two. Then within-group correlations are $1$ and $−1$ but the overall model has $R^2$ of 0.

A picture says it better. enter image description here

The orange points A and the blue points B both define $R^2$ of 1, quite different from the combined pattern.

My examples are contrived and extreme but clearly I can't see your data to experiment. But your groups don't have to be very different from each other for the effect you observed. Typically a group is more homogeneous (has less scatter) than the whole unless the grouping has nothing at all to do with the variables in question.

This effect is generic for regressions and similar analyses.

More interesting examples are often discussed under headings such as Simpson's paradox. A more direct term would be amalgamation paradox to describe the fact that collective patterns often seem to contradict subgroup patterns.

It is also entirely possible to get the opposite effect, a combined result stronger than any group result, but this is less common in practice.

$\endgroup$
  • $\begingroup$ +1 variation in such subgroup patterns means that an "average" pattern fitted to all of them will not usually fit nearly as well as the individual patterns would. $\endgroup$ – Glen_b Oct 24 '18 at 1:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.