How are regression, the t-test, and the ANOVA all versions of the general linear model?

Question

How are they all versions of the same basic statistical method?

Answer 1

Consider that they can all be written as a regression equation (perhaps with slightly differing interpretations than their traditional forms).

Regression: $$ Y=\beta_0 + \beta_1X_{\text{(continuous)}} + \varepsilon \\ \text{where }\varepsilon\sim\mathcal N(0, \sigma^2) $$

t-test: $$ Y=\beta_0 + \beta_1X_{\text{(dummy code)}} + \varepsilon \\ \text{where }\varepsilon\sim\mathcal N(0, \sigma^2) $$

ANOVA: $$ Y=\beta_0 + \beta_1X_{\text{(dummy code)}} + \varepsilon \\ \text{where }\varepsilon\sim\mathcal N(0, \sigma^2) $$

The prototypical regression is conceptualized with $X$ as a continuous variable. However, the only assumption that is actually made about $X$ is that it is a vector of known constants. It could be a continuous variable, but it could also be a dummy code (i.e., a vector of $0$'s & $1$'s that indicates whether an observation is a member of an indicated group--e.g., a treatment group). Thus, in the second equation, $X$ could be such a dummy code, and the p-value would be the same as that from a t-test in its more traditional form.

The meaning of the betas would differ here, though. In this case, $\beta_0$ would be the mean of the control group (for which the entries in the dummy variable would be $0$'s), and $\beta_1$ would be the difference between the mean of the treatment group and the mean of the control group.

Now, remember that it is perfectly reasonable to have / run an ANOVA with only two groups (although a t-test would be more common), and you have all three connected. If you prefer seeing how it would work if you had an ANOVA with 3 groups; it would be: $$ Y=\beta_0 + \beta_1X_{\text{(dummy code 1)}} + \beta_2X_{\text{(dummy code 2)}} + \varepsilon \\ \text{where }\varepsilon\sim\mathcal N(0, \sigma^2) $$ Note that when you have $g$ groups, you have $g-1$ dummy codes to represent them. The reference group (typically the control group) is indicated by having $0$'s for all dummy codes (in this case, both dummy code 1 & dummy code 2). In this case, you would not want to interpret the p-values of the t-tests for these betas that come with standard statistical output--they only indicate whether the indicated group differs from the control group when assessed in isolation. That is, these tests are not independent. Instead, you would want to assess whether the group means vary by constructing an ANOVA table and conducting an F-test. For what it's worth, the betas are interpreted just as with the t-test version described above: $\beta_0$ is the mean of the control / reference group, $\beta_1$ indicates the difference between the means of group 1 and the reference group, and $\beta_2$ indicates the difference between group 2 and the reference group.

---

In light of @whuber's comments below, these can also be represented via matrix equations:
$$ \bf Y=\bf X\boldsymbol\beta + \boldsymbol\varepsilon $$ Represented this way, $\bf Y$ & $\boldsymbol\varepsilon$ are vectors of length $N$, and $\boldsymbol\beta$ is a vector of length $p+1$. $\bf X$ is now a matrix with $N$ rows and $(p+1)$ columns. In a prototypical regression you have $p$ continuous $X$ variables and the intercept. Thus, your $\bf X$ matrix is composed of a series of column vectors side by side, one for each $X$ variable, with a column of $1$'s on the far left for the intercept.

If you are representing an ANOVA with $g$ groups in this way, remember that you would have $g-1$ dummy variables indicating the groups, with the reference group indicated by an observation having $0$'s in each dummy variable. As above, you would still have an intercept. Thus, $p=g-1$.

Answer 2

They can all be written as particular cases of the general linear model.

The t-test is a two-sample case of ANOVA. If you square the t-test statistic you get the corresponding $F$ in the ANOVA.

An ANOVA model is basically just a regression model where the factor levels are represented by dummy (or indicator) variables.

So if the model for a t-test is a subset of the ANOVA model and ANOVA is a subset of the multiple regression model, regression itself (and other things besides regression) is a subset of the general linear model, which extends regression to a more general specification of the error term than the usual regression case (which is 'independent' and 'equal-variance'), and to multivariate $Y$.

---

Here's an example showing the equivalence of the ordinary (equal-variance) two sample-$t$ analysis and a hypothesis test in a regression model, done in R (the actual data looks to be paired, so this isn't really a suitable analysis):

> t.test(extra ~ group, var.equal=TRUE, data = sleep) 

    Two Sample t-test

data:  extra by group
t = -1.8608, df = 18, p-value = 0.07919   
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -3.363874  0.203874
sample estimates:
mean in group 1 mean in group 2 
           0.75            2.33 

Note the p-value of 0.079 above. Here's the one way anova:

> summary(aov(extra~group,sleep))
            Df Sum Sq Mean Sq F value Pr(>F)  
group        1  12.48  12.482   3.463 0.0792 
Residuals   18  64.89   3.605 

Now for the regression:

> summary(lm(extra ~ group, data = sleep))

(some output removed)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)   0.7500     0.6004   1.249   0.2276  
group2        1.5800     0.8491   1.861   0.0792 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.899 on 18 degrees of freedom
Multiple R-squared:  0.1613,    Adjusted R-squared:  0.1147 
F-statistic: 3.463 on 1 and 18 DF,  p-value: 0.07919

Compare the p-value in the 'group2' row, and also the p-value for the F-test in the last row. For a two-tailed test, these are the same and both match the t-test result.

Further, the coefficient for 'group2' represents the difference in means for the two groups.

Finally note that $F=t^2$ for the statistics; this is also a fact about the reference distributions (that $t^2$ is $F$).

Answer 3

This answer that I posted earlier is somewhat relevant, but this question is somewhat different.

You might want to think about the differences and similarities between the following linear models: $$ \begin{bmatrix} Y_1 \\ \vdots \\ Y_n \end{bmatrix} = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ 1 & x_3 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix} \begin{bmatrix} \alpha_0 \\ \alpha_1 \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \vdots \\ \varepsilon_n \end{bmatrix} $$ $$ \begin{bmatrix} Y_1 \\ \vdots \\ Y_n \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 & \cdots & 0 \\ \vdots & \vdots & \vdots & & \vdots \\ 1 & 0 & 0 & \cdots & 0 \\ \hline 0 & 1 & 0 & \cdots & 0 \\ \vdots & \vdots & \vdots & & \vdots \\ 0 & 1 & 0 & \cdots & 0 \\ \hline 0 & 0 & 1 & \cdots & 0 \\ \vdots & & & & \vdots \\ \vdots & & & & \vdots \end{bmatrix} \begin{bmatrix} \alpha_0 \\ \vdots \\ \alpha_k \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \vdots \\ \varepsilon_n \end{bmatrix} $$

Answer 4

Anova is similar to a t-test for equality of means under the assumption of unknown but equal variances among treatments. This is because in ANOVA MSE is identical to pooled-variance used in t-test. There are other versions of t-test such as one for un-equal variances and pair-wise t-test. From this view, t-test can be more flexible.