12
$\begingroup$

I wonder what differences are between t-test and ANOVA in linear regression?

  1. Is a t-test to test whether any one of the slopes and intercept has mean zero, while ANOVA to test whether all slopes have mean zero? Is this the only difference between them?
  2. In simple linear regression i.e. where there is only one predictor variable, there is only one slope to estimate. So are t-test and ANOVA equivalent, and if yes, how, given that they are using different statistics (t-test is using t-statistic and ANOVA is using F-statistic)?
$\endgroup$
  • $\begingroup$ Ad 1) In linear regression, I normally understand ANOVA as a measure of goodness of fit of the model, i.e. to decide whether the model (regression line) explains substantial part of total variability. The question, whether it is equivalent to all slopes being zero, is really very interesting. Ad 2) it looks I'm getting almost the same p-values for t-test and regression ANOVA in this case. Really interesting theorem! $\endgroup$ – Curious Oct 13 '11 at 16:49
17
$\begingroup$

The general linear model lets us write an ANOVA model as a regression model. Let's assume we have two groups with two observations each, i.e., four observations in a vector $y$. Then the original, overparametrized model is $E(y) = X^{\star} \beta^{\star}$, where $X^{\star}$ is the matrix of predictors, i.e., dummy-coded indicator variables: $$ \left(\begin{array}{c}\mu_{1} \\ \mu_{1} \\ \mu_{2} \\ \mu_{2}\end{array}\right) = \left(\begin{array}{ccc}1 & 1 & 0 \\ 1 & 1 & 0 \\ 1 & 0 & 1 \\ 1 & 0 & 1\end{array}\right) \left(\begin{array}{c}\beta_{0}^{\star} \\ \beta_{1}^{\star} \\ \beta_{2}^{\star}\end{array}\right) $$

The parameters are not identifiable as $((X^{\star})' X^{\star})^{-1} (X^{\star})' E(y)$ because $X^{\star}$ has rank 2 ($(X^{\star})'X^{\star}$ is not invertible). To change that, we introduce the constraint $\beta_{1}^{\star} = 0$ (treatment contrasts), which gives us the new model $E(y) = X \beta$: $$ \left(\begin{array}{c}\mu_{1} \\ \mu_{1} \\ \mu_{2} \\ \mu_{2}\end{array}\right) = \left(\begin{array}{cc}1 & 0 \\ 1 & 0 \\ 1 & 1 \\ 1 & 1\end{array}\right) \left(\begin{array}{c}\beta_{0} \\ \beta_{2}\end{array}\right) $$

So $\mu_{1} = \beta_{0}$, i.e., $\beta_{0}$ takes on the meaning of the expected value from our reference category (group 1). $\mu_{2} = \beta_{0} + \beta_{2}$, i.e., $\beta_{2}$ takes on the meaning of the difference $\mu_{2} - \mu_{1}$ to the reference category. Since with two groups, there is just one parameter associated with the group effect, the ANOVA null hypothesis (all group effect parameters are 0) is the same as the regression weight null hypothesis (the slope parameter is 0).

A $t$-test in the general linear model tests a linear combination $\psi = \sum c_{j} \beta_{j}$ of the parameters against a hypothesized value $\psi_{0}$ under the null hypothesis. Choosing $c = (0, 1)'$, we can thus test the hypothesis that $\beta_{2} = 0$ (the usual test for the slope parameter), i.e. here, $\mu_{2} - \mu_{1} = 0$. The estimator is $\hat{\psi} = \sum c_{j} \hat{\beta}_{j}$, where $\hat{\beta} = (X'X)^{-1} X' y$ are the OLS estimates for the parameters. The general test statistic for such $\psi$ is: $$ t = \frac{\hat{\psi} - \psi_{0}}{\hat{\sigma} \sqrt{c' (X'X)^{-1} c}} $$

$\hat{\sigma}^{2} = \|e\|^{2} / (n-\mathrm{Rank}(X))$ is an unbiased estimator for the error variance, where $\|e\|^{2}$ is the sum of the squared residuals. In the case of two groups $\mathrm{Rank}(X) = 2$, $(X'X)^{-1} X' = \left(\begin{smallmatrix}.5 & .5 & 0 & 0 \\-.5 & -.5 & .5 & .5\end{smallmatrix}\right)$, and the estimators thus are $\hat{\beta}_{0} = 0.5 y_{1} + 0.5 y_{2} = M_{1}$ and $\hat{\beta}_{2} = -0.5 y_{1} - 0.5 y_{2} + 0.5 y_{3} + 0.5 y_{4} = M_{2} - M_{1}$. With $c' (X'X)^{-1} c$ being 1 in our case, the test statistic becomes: $$ t = \frac{M_{2} - M_{1} - 0}{\hat{\sigma}} = \frac{M_{2} - M_{1}}{\sqrt{\|e\|^{2} / (n-2)}} $$

$t$ is $t$-distributed with $n - \mathrm{Rank}(X)$ df (here $n-2$). When you square $t$, you get $\frac{(M_{2} - M_{1})^{2} / 1}{\|e\|^{2} / (n-2)} = \frac{SS_{b} / df_{b}}{SS_{w} / df_{w}} = F$, the test statistic from the ANOVA $F$-test for two groups ($b$ for between, $w$ for within groups) which follows an $F$-distribution with 1 and $n - \mathrm{Rank}(X)$ df.

With more than two groups, the ANOVA hypothesis (all $\beta_{j}$ are simultaneously 0, with $1 \leq j$) refers to more than one parameter and cannot be expressed as a linear combination $\psi$, so then the tests are not equivalent.

$\endgroup$
3
$\begingroup$

In 1, ANOVA will usually test factor variables and whether or not between group variance is significant. You'll clearly see the difference if your software allows indicator variables in a regression: for each dummy you'll get a p value saying whether this group scores significantly different from 0, and as a consequence significantly different than the reference group or reference value applicable. Usually, you won't see to what degree the indicator itself is important until you do an ANOVA test.

A F-test is a squared t-test. Therefore, in 2, it's the same.

$\endgroup$
  • $\begingroup$ Thanks! (1) What do indicator variables mean here? (2) Generally, a t-test is equivalent to ANOVA only when there are only two groups. But in simple linear regression there may be more than two groups, where the number of groups is the number of values the predictor variable takes in the data set. $\endgroup$ – Tim Oct 13 '11 at 9:13
  • $\begingroup$ (1) Indicator or categorical or factor variable ... all the same. (2) Indeed, but you may want to know how well a set of dummies/categories scores from ANOVA. $\endgroup$ – Labour Oct 13 '11 at 9:33
  • $\begingroup$ Thanks! (2) So in simple linear regression, how is t-test equivalent to ANOVA, given that there are more than two groups? What does "how well a set of dummies/categories scores from ANOVA" mean, and why do I want to know it? $\endgroup$ – Tim Oct 13 '11 at 9:49
  • $\begingroup$ In OLS regression, R² (explained variance) will be equal to eta² or MSS / TSS from ANOVA no matter how many groups you define. Next, you might want to know the contribution of a set of dummies (i.e. an indicator variable) to say whether the set itself is relevant and to what extent, which is different from the significance of the difference between one single category with the reference category. $\endgroup$ – Labour Oct 13 '11 at 11:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.