50
$\begingroup$

How are they all versions of the same basic statistical method?

$\endgroup$
48
$\begingroup$

Consider that they can all be written as a regression equation (perhaps with slightly differing interpretations than their traditional forms).

Regression: $$ Y=\beta_0 + \beta_1X_{\text{(continuous)}} + \varepsilon \\ \text{where }\varepsilon\sim\mathcal N(0, \sigma^2) $$

t-test: $$ Y=\beta_0 + \beta_1X_{\text{(dummy code)}} + \varepsilon \\ \text{where }\varepsilon\sim\mathcal N(0, \sigma^2) $$

ANOVA: $$ Y=\beta_0 + \beta_1X_{\text{(dummy code)}} + \varepsilon \\ \text{where }\varepsilon\sim\mathcal N(0, \sigma^2) $$

The prototypical regression is conceptualized with $X$ as a continuous variable. However, the only assumption that is actually made about $X$ is that it is a vector of known constants. It could be a continuous variable, but it could also be a dummy code (i.e., a vector of $0$'s & $1$'s that indicates whether an observation is a member of an indicated group--e.g., a treatment group). Thus, in the second equation, $X$ could be such a dummy code, and the p-value would be the same as that from a t-test in its more traditional form.

The meaning of the betas would differ here, though. In this case, $\beta_0$ would be the mean of the control group (for which the entries in the dummy variable would be $0$'s), and $\beta_1$ would be the difference between the mean of the treatment group and the mean of the control group.

Now, remember that it is perfectly reasonable to have / run an ANOVA with only two groups (although a t-test would be more common), and you have all three connected. If you prefer seeing how it would work if you had an ANOVA with 3 groups; it would be: $$ Y=\beta_0 + \beta_1X_{\text{(dummy code 1)}} + \beta_2X_{\text{(dummy code 2)}} + \varepsilon \\ \text{where }\varepsilon\sim\mathcal N(0, \sigma^2) $$ Note that when you have $g$ groups, you have $g-1$ dummy codes to represent them. The reference group (typically the control group) is indicated by having $0$'s for all dummy codes (in this case, both dummy code 1 & dummy code 2). In this case, you would not want to interpret the p-values of the t-tests for these betas that come with standard statistical output--they only indicate whether the indicated group differs from the control group when assessed in isolation. That is, these tests are not independent. Instead, you would want to assess whether the group means vary by constructing an ANOVA table and conducting an F-test. For what it's worth, the betas are interpreted just as with the t-test version described above: $\beta_0$ is the mean of the control / reference group, $\beta_1$ indicates the difference between the means of group 1 and the reference group, and $\beta_2$ indicates the difference between group 2 and the reference group.


In light of @whuber's comments below, these can also be represented via matrix equations:
$$ \bf Y=\bf X\boldsymbol\beta + \boldsymbol\varepsilon $$ Represented this way, $\bf Y$ & $\boldsymbol\varepsilon$ are vectors of length $N$, and $\boldsymbol\beta$ is a vector of length $p+1$. $\bf X$ is now a matrix with $N$ rows and $(p+1)$ columns. In a prototypical regression you have $p$ continuous $X$ variables and the intercept. Thus, your $\bf X$ matrix is composed of a series of column vectors side by side, one for each $X$ variable, with a column of $1$'s on the far left for the intercept.

If you are representing an ANOVA with $g$ groups in this way, remember that you would have $g-1$ dummy variables indicating the groups, with the reference group indicated by an observation having $0$'s in each dummy variable. As above, you would still have an intercept. Thus, $p=g-1$.

$\endgroup$
  • 1
    $\begingroup$ The ANOVA equation would make sense as an ANOVA (and not a t-test) only if $\beta_1$ were interpreted as a vector and multiplied on the right. $\endgroup$ – whuber May 15 '13 at 13:17
  • $\begingroup$ These aren't matrix equations; I rarely use those here, as many people don't read them. The 1st ANOVA represents an identical situation as the preceding t-test. I'm just pointing out that if you can run a 2-sample independent t-test, you can run the same data as an ANOVA (which many people should recognize / remember from their stats 101 class). I add another ANOVA version w/ 3 groups lower down to clarify that a 2-group situation isn't the only ANOVA case that can be understood as a regression; but the reg equation now looks different--I was trying to maintain a more explicit parallel above. $\endgroup$ – gung - Reinstate Monica May 15 '13 at 13:47
  • $\begingroup$ My point is that unless you do make it a matrix equation, your characterization of ANOVA is too limited to be useful: it is identical to your characterization of the t-test and so is more confusing than it is helpful. When you start introducing more groups, you suddenly change the equation, which may also be less than clear. Whether you want to use matrix notation is of course up to you, but in the interest of communicating well you should strive for consistency. $\endgroup$ – whuber May 15 '13 at 13:49
  • $\begingroup$ Could you please explain a bit more on how you arrive from popular definition of t-test to the equation you have shown.Basically I can't figure out what is Y here (it could be naivity or less IQ for stats). However how to arrive from t = (y-x-u0)/s to this equation. $\endgroup$ – Gaurav Singhal Jul 29 '16 at 21:19
  • $\begingroup$ It doesn't, although this may be unfamiliar to you. $Y$ is continuous (& assumed conditionally normal) in all cases listed. There are no distributional assumptions about $X$, it can be continuous, dichotomous, or a multi-level categorical variable. $\endgroup$ – gung - Reinstate Monica Oct 29 '16 at 15:56
17
$\begingroup$

They can all be written as particular cases of the general linear model.

The t-test is a two-sample case of ANOVA. If you square the t-test statistic you get the corresponding $F$ in the ANOVA.

An ANOVA model is basically just a regression model where the factor levels are represented by dummy (or indicator) variables.

So if the model for a t-test is a subset of the ANOVA model and ANOVA is a subset of the multiple regression model, regression itself (and other things besides regression) is a subset of the general linear model, which extends regression to a more general specification of the error term than the usual regression case (which is 'independent' and 'equal-variance'), and to multivariate $Y$.


Here's an example showing the equivalence of the ordinary (equal-variance) two sample-$t$ analysis and a hypothesis test in a regression model, done in R (the actual data looks to be paired, so this isn't really a suitable analysis):

> t.test(extra ~ group, var.equal=TRUE, data = sleep) 

    Two Sample t-test

data:  extra by group
t = -1.8608, df = 18, p-value = 0.07919   
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -3.363874  0.203874
sample estimates:
mean in group 1 mean in group 2 
           0.75            2.33 

Note the p-value of 0.079 above. Here's the one way anova:

> summary(aov(extra~group,sleep))
            Df Sum Sq Mean Sq F value Pr(>F)  
group        1  12.48  12.482   3.463 0.0792 
Residuals   18  64.89   3.605                 

Now for the regression:

> summary(lm(extra ~ group, data = sleep))

(some output removed)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)   0.7500     0.6004   1.249   0.2276  
group2        1.5800     0.8491   1.861   0.0792 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.899 on 18 degrees of freedom
Multiple R-squared:  0.1613,    Adjusted R-squared:  0.1147 
F-statistic: 3.463 on 1 and 18 DF,  p-value: 0.07919

Compare the p-value in the 'group2' row, and also the p-value for the F-test in the last row. For a two-tailed test, these are the same and both match the t-test result.

Further, the coefficient for 'group2' represents the difference in means for the two groups.

$\endgroup$
  • $\begingroup$ Having same p values in all 3 scenarios is magical and impressive, however if you could explain a bit more on how these p-values gets calculated, it would definitely make this answer more interesting. I don't know if showing p-value calculations will make it more useful as well, so that is something you could decide. $\endgroup$ – Gaurav Singhal Jul 29 '16 at 20:49
  • $\begingroup$ @Gaurav The p-values are the same because you're testing the same hypothesis on the same model, just represented slightly differently. If you're interested in how some specific p-value is calculated, it would be a new question (it would not be an answer to the question here). You're free to ask such a question though try a search first since it may already have been answered. $\endgroup$ – Glen_b -Reinstate Monica Jul 30 '16 at 2:43
  • $\begingroup$ Thanks @Glen_b, sorry for asking an obvious question and that too in not the best way. And you still answered my question - "same hypothesis on the same model (and/or data)". I did not give enough thoughts on how they are testing the same hypothesis. Thanks $\endgroup$ – Gaurav Singhal Jul 31 '16 at 19:31
2
$\begingroup$

This answer that I posted earlier is somewhat relevant, but this question is somewhat different.

You might want to think about the differences and similarities between the following linear models: $$ \begin{bmatrix} Y_1 \\ \vdots \\ Y_n \end{bmatrix} = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ 1 & x_3 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix} \begin{bmatrix} \alpha_0 \\ \alpha_1 \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \vdots \\ \varepsilon_n \end{bmatrix} $$ $$ \begin{bmatrix} Y_1 \\ \vdots \\ Y_n \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 & \cdots & 0 \\ \vdots & \vdots & \vdots & & \vdots \\ 1 & 0 & 0 & \cdots & 0 \\ \hline 0 & 1 & 0 & \cdots & 0 \\ \vdots & \vdots & \vdots & & \vdots \\ 0 & 1 & 0 & \cdots & 0 \\ \hline 0 & 0 & 1 & \cdots & 0 \\ \vdots & & & & \vdots \\ \vdots & & & & \vdots \end{bmatrix} \begin{bmatrix} \alpha_0 \\ \vdots \\ \alpha_k \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \vdots \\ \varepsilon_n \end{bmatrix} $$

$\endgroup$
  • 2
    $\begingroup$ Some description and comment to the questions would useful for the readers since now they have to guess where did they came from and how do they relate to the question... $\endgroup$ – Tim Nov 10 '15 at 12:24
0
$\begingroup$

Anova is similar to a t-test for equality of means under the assumption of unknown but equal variances among treatments. This is because in ANOVA MSE is identical to pooled-variance used in t-test. There are other versions of t-test such as one for un-equal variances and pair-wise t-test. From this view, t-test can be more flexible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.