11
$\begingroup$

I have two regressions of the same Y and three-level X. Overall n=15, with n=5 in each group or level of X. The first regression treats the X as categorical, assigning indicator variables to levels 2 and 3 with level one being the reference. Indicators / dummies are like so: X1 = 1 if level = 2, 0 if else X2 = 1 if level = 3, 0 if else

As a result my fitted model looks something like this: y = b0 + b1(x1) + b2(x2)

I run the regression, and the output includes this Analysis of Variance table:

table

The rest of the output is irrelevant here.

Okay so now I run a different regression on the same data. I ditch the categorical analysis and treat X as continuous, but I add a variable to the equation: X^2, the square of X. So now I have the following model: y = b0 + b1(X) + b2(X)^2

If I run it, it spits out the same exact Analysis of Variance table that I showed you above. Why do these two regressions give rise to the same tables?

[Credit for this little conundrum goes to Thomas Belin in the Dept of Biostatistics at the University of California Los Angeles.]

$\endgroup$
  • $\begingroup$ I think you're gonna have to show us the code that "does the regression" and possibly the data step (looks like SAS output to me) you use to create the data tabel upon which you're operating. $\endgroup$ – Brad S. Dec 6 '17 at 23:00
  • 1
    $\begingroup$ @Brad I don't think that's necessary: the situation is clearly described and no more information is needed to explain what's going on. $\endgroup$ – whuber Dec 6 '17 at 23:13
  • $\begingroup$ @whuber Maybe. I guess, if you say so but it feels like a programming error to me. I look forward to your answer. $\endgroup$ – Brad S. Dec 6 '17 at 23:30
  • 1
    $\begingroup$ @Brad Not a programming error: I posted my explanation. It's a good question, with genuine statistical interest (and applicability). $\endgroup$ – whuber Dec 6 '17 at 23:40
  • $\begingroup$ Hey Brad, It's actually from a problem set - the situation was given to me much the same way I gave it to you guys, and the question kinda posed the same way: "why would they be the same?". It's just how I lay it out: two models, same ANOVA tables, rest of the outputs not even given (I should've made that clear instead of saying "irrelevant"). $\endgroup$ – logjammin Dec 6 '17 at 23:41
21
$\begingroup$

In matrix terms your models are in the usual form $E[Y]=X\beta$.

The first model represents an element of the first group by the row $(1,0,0)$ in $X$, corresponding to the intercept, the indicator for category 2, and the indicator for category 3. It represents an element of the second group by the row $(1,1,0)$ and an element of the third group by $(1,0,1)$.

The second model instead uses rows $(1,1,1^2)=(1,1,1)$, $(1,2,2^2)=(1,2,4)$, and $(1,3,3^2)=(1,3,9)$, respectively.

Let's call the resulting model matrices $X_1$ and $X_2$. They are simply related: the columns of one are linear combinations of the columns of the other. For instance, let

$$V = \pmatrix{1&1&1 \\ 0&1&3 \\ 0&2&8}.$$

Then since

$$\pmatrix{1&0&0 \\ 1&1&0 \\ 1&0&1} V = \pmatrix{1&1&1 \\ 1&2&4 \\ 1&3&9},$$

it follows that

$$X_1 V = X_2.$$

The models themselves therefore are related by

$$X_1\beta_1 = E[Y] = X_2\beta_2 = (X_1V)\beta_2 = X_1(V\beta_2).$$

That is, the coefficients $\beta_2$ for the second model must be related to those of the first one via

$$\beta_1 = V\beta_2.$$

The same relationship therefore holds for their least squares estimates. This shows that the models have identical fits: they merely express them differently.

Since the first columns of the two model matrices are the same, any ANOVA table that decomposes variance between the first column and the remaining columns will not change. An ANOVA table that distinguishes between the second and third columns, though, will depend on how the data are encoded.

Geometrically (and somewhat more abstractly), the three-dimensional subspace of $\mathbb{R}^{15}$ generated by the columns of $X_1$ coincides with the subspace generated by the columns of $X_2$. Therefore the models will have identical fits. The fits are expressed differently only because the spaces are described with two different bases.


To illustrate, here are data just like yours (but with different responses) and the corresponding analyses as generated in R.

set.seed(17)
D <- data.frame(group=rep(1:3, each=5), y=rnorm(3*5, rep(1:3, each=5), sd=2))

Fit the two models:

fit.1 <- lm(y ~ factor(group), D)
fit.2 <- lm(y ~ group + I(group^2), D)

Display their ANOVA tables:

anova(fit.1)
anova(fit.2)

The output for the first model is

              Df Sum Sq Mean Sq F value   Pr(>F)    
factor(group)  2 51.836  25.918  14.471 0.000634 ***
Residuals     12 21.492   1.791 

For the second model it is

           Df Sum Sq Mean Sq F value    Pr(>F)    
group       1 50.816  50.816 28.3726 0.0001803 ***
I(group^2)  1  1.020   1.020  0.5694 0.4650488    
Residuals  12 21.492   1.791  

You can see that the residual sums of squares are the same. By adding the first two rows in the second model you will obtain the same DF and sum of squares, from which the same mean square, F value, and p-value can be computed.

Finally, let's compare the coefficient estimates.

beta.1.hat <- coef(fit.1)
beta.2.hat <- coef(fit.2)

The output is

(Intercept) factor(group)2 factor(group)3 
  0.4508762      2.8073697      4.5084944 

(Intercept)       group  I(group^2) 
 -3.4627385   4.4667371  -0.5531225 

Even the intercepts are completely different. That's because the estimates of any variable in a multiple regression depend on the estimates of all other variables (unless they are all mutually orthogonal, which is not the case for either model). However, look at what multiplication by $V$ accomplishes:

$$\pmatrix{1&1&1 \\ 0&1&3 \\ 0&2&8}\pmatrix{-3.4627385 \\ 4.4667371 \\-0.5531225} = \pmatrix{ 0.4508762 \\ 2.8073697 \\ 4.5084944 }.$$

The fits really are the same just as claimed.

$\endgroup$
  • 5
    $\begingroup$ Holy smokes, man. I've never gotten more considered, thorough answer from asking the internet a question. Thank you x1000, seriously. $\endgroup$ – logjammin Dec 6 '17 at 23:47
  • $\begingroup$ Welcome to our site! I hope you continue using it and look forward to your contributions. $\endgroup$ – whuber Dec 6 '17 at 23:48
  • 1
    $\begingroup$ I learned something today! (upvoted) $\endgroup$ – Brad S. Dec 6 '17 at 23:53
  • $\begingroup$ Amazing answer. Mind blown! $\endgroup$ – kedarps Dec 7 '17 at 3:15
5
$\begingroup$

Briefly, both models are saturated in the sense that they provide unique empirical predictions of the response at all 3 levels of X. It may be obvious for the factor variable coding in model 1. For a quadratic trend, it is interesting to note that a quadratic formula can interpolate any 3 points. While the contrasts are different, in both models the global test against a null of an intercept only model provides identical inference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.