I have a data set with two columns, the first of which is to be used as a response variable and the second of which is to be used as a predictor variable. The predictor variable, however, is populated by integer values, many of which are zero. When I place data points for the predictor and response variables into groups corresponding groups, no group has a value of zero anymore. Performing a linear regression on the data when it is in this form produces a significantly higher $R^2$ than in the ungrouped regression. My question is: Is something seriously wrong with my methodology if the change in $R^2$ is massive between the grouped and ungrouped models? Can this be explained by the large number of zeros?

To provide a slightly better idea of what I mean by grouping consider the following data:

A  B

0  .2
0  .3
0  .3
2  .6
4  .7
6  .8
7  .7
4  .4

A would be grouped as follows:


with the new B values being the group averages.

  • 1
    $\begingroup$ It's a little unclear what models you're actually fitting. "Grouping" the response or the predictor(s)? What's a "basic regression"? This sounds like the sort of thing that can happen when the response varies in a non-linear fashion with a predictor. Perhaps you need to include polynomial or spline terms; perhaps an indicator variable for when the predictor is zero. What about the usual regression diagnostics? $\endgroup$ – Scortchi - Reinstate Monica Jun 17 '14 at 21:09
  • 1
    $\begingroup$ How are you determining the value of the explanatory variable for each group? Are you averaging them? $\endgroup$ – shadowtalker Jun 17 '14 at 21:15
  • $\begingroup$ @scortchi The usual diagnostics, in particular the effect size (using $R^2$) come out much better when the data is grouped as opposed to ungrouped. I did mean grouping both the response and predictor variables, I will adjust that in the question. $\endgroup$ – 114 Jun 17 '14 at 21:16
  • 1
    $\begingroup$ Plot of residuals vs fitted values? Could you write down the two models? And what do the variables represent? $\endgroup$ – Scortchi - Reinstate Monica Jun 17 '14 at 21:20
  • 1
    $\begingroup$ So you're averaging the response! The coefficient of determination's bound to change. $\endgroup$ – Scortchi - Reinstate Monica Jun 17 '14 at 21:57

It's still not completely clear what you're doing, but it sounds like it involves comparing the coefficients of determination ($R^2$s) for regression models of different responses— individual observations & groupwise averages— & drawing conclusions about the adequacy of the first model. This makes no sense. The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares

$$R^2=\frac{\sum_i (\hat y_i - \bar y)^2}{\sum_i (y_i - \bar y)^2}$$

where $y_i$ is the response for the $i$th observation. A model that changes the estimates $\hat y_i$ to be farther from the grand mean $\bar y$ increases the regression sum of squares at the expense of the residual sum of squares, thus explaining more of the variability in the response. A model that redefines every $y$ in the equation is simply incomparable. Predicting groupwise averages is a different task to predicting individual observations: & an easier one; you'd expect the lower variability in averages, & the reduced number of data points compared to estimated parameters, to result in better fit.

Your example in fact illustrates the issue: regressing $B$ on $A$ gives $R^2\approx 0.7$; regressing the groupwise averages of $B$ on the grouped values of $A$ gives $R^2=1$.

So there's no evidence that the model on individual observations is inadequate, & no reason to suppose that its inadequacy would be somehow hidden from the usual diagnostic methods.

There are no assumptions on the distribution of predictors in regression, so knowing there are many zeroes is at most a clue to the form of model worth considering, in a particular context. For example, if you wanted to predict the ferocity of a swan ($y$) from how many cygnets it has ($x_1$), it would make sense to treat the having some/having none distinction specially— by including a dummy variable $x_2$ in the model to indicate when the count of cygnets is exactly zero:

$$\operatorname {E} Y = \beta_0 + \beta_1 x_1 + \beta_2 x_2$$

where the $\beta$s are the coefficients to be estimated. The effect is to fit the usual regression line through non-zero counts of cygnets ($x_1>0 \Rightarrow x_2=0$)

$$\operatorname {E} Y = \beta_0 + \beta_1 x_1$$

& independently estimate ferocity for a zero count ($x_1=0\Rightarrow x_2=1)$

$$\operatorname {E} Y = (\beta_0 + \beta_2)$$

See How to Include an Independent Variable with one-half 0s, one-half non-0 values for a graphical illustration.

† For OLS fits with an intercept.

| cite | improve this answer | |

I am going where angels fear to tread answering after a good answer has been posted.

I think that you are thinking about a classification and regression tree model (CART model) as an approximation. You didn't say that, but splitting into subsets and using the average of each group as an approximation is a CART model.

So here is your raw data. enter image description here

If we approximate it with its mean, then we get the data, the mean approximation, and the error as follows.

enter image description here

You can see that the mean error is zero and that the count of positive errors is equal to the count of negative errors.

Now a CART model assumes that I want to split this and looks for the best split point. It wants to minimize the "error" by splitting at a meaningful point. If it pops out one of the values with low error, but leaves the other alone, then it might not really change the error to much.

If you don't look at it as "input-output" then you can think in terms of partitioning on x, on y, or along some mixture of the two in order to reduce the error in the approximation. "Error in the approximation", at this point, is too imprecise to inform the next step.

The textbook measure of variation is "variance".

$Var \left ( X \right) = E \left[ \left( X - E \left[ X \right] \right)^2\right]$

The ($R^2$), or "coefficient of determination" is a ratio of two variances on equal sample sizes subtracted from 1.

$R^2 = 1 - \frac {E \left[ \left( X - f \left[ X \right] \right)^2\right]} {E \left[ \left( X - E \left[ X \right] \right)^2\right]} $

Inspection shows that as the numerator variance goes to zero the coefficient of determination goes to unity. For a reduction of variation to zero, determination goes to one. Inspection also shows that as the numerator variance approaches the population variance the coefficient of determination goes to zero. This means that a fit as poor as the raw (or worse) will have a lower variance.


Here are my results for the linear fit

row x   y       f(x|all)    (y-f(y))2   (y-E(y))2   Ratio
1   0   0.2     0.30        0.0102      0.090       0.113
2   0   0.3     0.30        0.0000      0.040       0.000
3   0   0.3     0.30        0.0000      0.040       0.000
4   2   0.6     0.44        0.0258      0.010       2.579
5   4   0.7     0.58        0.0149      0.040       0.373
6   6   0.8     0.72        0.0070      0.090       0.078
7   7   0.7     0.79        0.0073      0.040       0.183
8   4   0.4     0.58        0.0316      0.010       3.165

From these I compute the numerator to be 0.0969, the denominator to be 0.3600, the ratio to be 0.2690, and the ($R^2$) to be 73.10%.

The row-ratio gives an indicator of which values have higher variation when compared to the fit than when compared to the mean. Rows 4 and 8 are the second and first ranked in terms of ratio. A value below one is going to be better represented by the fit line. A value above one is going to be better represented by the ensemble average. Splitting out the first three rows is removing three values that are better fit by the fit line than the ensemble average.


Now you want to split and when I hear that I do not hear "cull". When I hear this I hear that you are changing the f(x) into a piecewise fit. For x>0 you will have one function and for x=0 you have another.

When I use that approach I get the following data:

row x   y   C       f(x|C)  (y-f(y|C))^2    (y-E(y))^2  Ratio
1   0   0.2 1       0.27    0.0044          0.090       0.049
2   0   0.3 1       0.27    0.0011          0.040       0.028
3   0   0.3 1       0.27    0.0011          0.040       0.028
4   2   0.6 2       0.54    0.0035          0.010       0.351
5   4   0.7 2       0.62    0.0069          0.040       0.172
6   6   0.8 2       0.69    0.0114          0.090       0.126
7   7   0.7 2       0.73    0.0010          0.040       0.025
8   4   0.4 2       0.62    0.0471          0.010       4.713

For this updated set I compute the numerator as 0.0765, the denominator as 0.3600, the ratio as 0.2126, and the ($R^2$) as 78.74%.

Looking at the "ratio" values, the row 4 went from 2.57 to 0.351, but row 8 went from 3.165 to 4.713. In the first case there was an improvement, but in the second there was disimprovement. (Not sure what the technically appropriate term is here.) This new model has more parameters, and better predicts your data.

Now if you wanted to then you could use piecewise linear interpolation and fit the data exactly, but in applied models the goal is "generalization". This means that you want to fit the "signal" and not the "noise". A linear interpolation often fits the noise.

Looking at CART approach

In an orthogonal CART approach you want to sweep over the domain looking for splits that improve the coefficient of determination. The assumed model is an average, not a linear fit. A binary split is the simplest to model.

Here is an example data table for a binary partition at x=1.

row x    y  C       f(x|C)  (y-f(y|C))^2    (y-E(y))^2  Ratio
1   0   0.2 1       0.27    0.0044          0.090       0.049
2   0   0.3 1       0.27    0.0011          0.040       0.028
3   0   0.3 1       0.27    0.0011          0.040       0.028
4   2   0.6 2       0.64    0.0016          0.010       0.160
5   4   0.7 2       0.64    0.0036          0.040       0.090
6   6   0.8 2       0.64    0.0256          0.090       0.284
7   7   0.7 2       0.64    0.0036          0.040       0.090
8   4   0.4 2       0.64    0.0576          0.010       5.760

For your data there are 5 split locations: c([1,3,5,6.5]).

Here are the results at each of those split locations:

Split   num     den     ratio   R^2     comment
-1      0.3600  0.3600  1.0000  0.0000  mean fit
1       0.0987  0.3600  0.2741  0.7259  proposed split
3       0.1800  0.3600  0.5000  0.5000  
5       0.1933  0.3600  0.5370  0.4630  
6.5     0.3143  0.3600  0.8730  0.1270  

This says that if you were using a piecewise mean, not a linear fit, as the approximating function that the best improvement was the one that you suggested.

The coefficient of determination for these piecewise constant models is nearly as good as the linear fit, and but the parameter counts are not equal. For the binary piecewise constant models you have the two means plus the split location for a parameter count of 3. For the linear model you have only the slope and the y-intercept.

If we split the piecewise function at 1.0 and use linear models instead of piecewise constant models, then the results, if you examine the previous tables, beats the ensemble linear fit.

For the selected piecewise linear model the numerator was 0.0765, the denominator was 0.3600, the ratio was 0.2126, and the ($R^2$) was 78.74%. This compared favorably against the non-piecewise linear fit ($R^2$) of 73.10%. It was a 7.7% improvement.

Inspection shows that the 8th row, point (4,0.4) has the highest variance and is most poorly described by the model.

If the data were split according to the following image, then the coefficient of determination might be even better, but that exploration is outside the scope of the question.

enter image description here

If we group then fit as suggested then we get the following:

row x   y   C       f(x|C)  (y-f(y|C))^2    (y-E(y))^2  Ratio
1   0   0.2 1       0.20    0.0000          0.090       0.000
8   4   0.4 1       0.40    0.0000          0.010       0.000
2   0   0.3 2       0.36    0.0035          0.040       0.088
3   0   0.3 2       0.36    0.0035          0.040       0.088
4   2   0.6 2       0.49    0.0120          0.010       1.203
5   4   0.7 2       0.62    0.0062          0.040       0.155
6   6   0.8 2       0.75    0.0023          0.090       0.026
7   7   0.7 2       0.82    0.0138          0.040       0.345

In this case the computed numerator is 0.0414, the denominator is 0.36, the ratio is 0.1150, and the ($R^2$) is 88.50%. Further gains might be made by approximating group "2" using a quadratic or other curve.

Best of Luck.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.