103
$\begingroup$

ANOVA is equivalent to linear regression with the use of suitable dummy variables. The conclusions remain the same irrespective of whether you use ANOVA or linear regression.

In light of their equivalence, is there any reason why ANOVA is used instead of linear regression?

Note: I am particularly interested in hearing about technical reasons for the use of ANOVA instead of linear regression.

Edit

Here is one example using one-way ANOVA. Suppose, you want to know if the average height of male and females is the same. To test for your hypothesis you would collect data from a random sample of male and females (say 30 each) and perform the ANOVA analysis (i.e., sum of squares for sex and error) to decide whether an effect exists.

You could also use linear regression to test for this as follows:

Define: $\text{Sex} = 1$ if respondent is a male and $0$ otherwise. $$ \text{Height} = \text{Intercept} + \beta * \text{Sex} + \text{error} $$ where: $\text{error}\sim\mathcal N(0,\sigma^2)$

Then a test of whether $\beta = 0$ is a an equivalent test for your hypothesis.

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ If I am not mistaken, linear regression is the estimation of coefficients that define a good linear map from X to Y. ANOVA is a test to know if there is significant differences in X when Y take two different values. Can you explain us why you think they are the same? $\endgroup$
    – robin girard
    Jul 23, 2010 at 15:29
  • 31
    $\begingroup$ ANOVA can be seen as "syntactic sugar" for a special subgroup of linear regression models. ANOVA is regularly used by researchers who are not statisticians by training. They are now "institutionalized" and its hard to convert them back to using the more general representation ;-) $\endgroup$
    – suncoolsu
    Oct 14, 2010 at 16:52
  • 6
    $\begingroup$ Upvoted your comment, but experimentalists are even crazier than I thought if this is syntactic sugar for them! Which version is more intuitive.... ANOVA hypothesis test on $\beta$: is the ratio of explained variance to the unexplained variance sufficiently high? T-test on the $\beta$ term of a regression model: is the effect of $\beta$ sufficiently different from zero? And, with the latter formulation you also get the direction of change. And, if you had to transform the data, you can back-transform the parameter estimate into a physically meaningful quantity. Unlike SS. $\endgroup$
    – f1r3br4nd
    Sep 25, 2013 at 20:44
  • $\begingroup$ See also stats.stackexchange.com/questions/268006/… $\endgroup$
    – kjetil b halvorsen ♦
    Dec 10, 2019 at 10:58

5 Answers 5

Reset to default
62
$\begingroup$

As an economist, the analysis of variance (ANOVA) is taught and usually understood in relation to linear regression (e.g. in Arthur Goldberger's A Course in Econometrics). Economists/Econometricians typically view ANOVA as uninteresting and prefer to move straight to regression models. From the perspective of linear (or even generalised linear) models, ANOVA assigns coefficients into batches, with each batch corresponding to a "source of variation" in ANOVA terminology.

Generally you can replicate the inferences you would obtain from ANOVA using regression but not always OLS regression. Multilevel models are needed for analysing hierarchical data structures such as "split-plot designs," where between-group effects are compared to group-level errors, and within-group effects are compared to data-level errors. Gelman's paper [1] goes into great detail about this problem and effectively argues that ANOVA is an important statistical tool that should still be taught for it's own sake.

In particular Gelman argues that ANOVA is a way of understanding and structuring multilevel models. Therefore ANOVA is not an alternative to regression but as a tool for summarizing complex high-dimensional inferences and for exploratory data analysis.

Gelman is a well-respected statistician and some credence should be given to his view. However, almost all of the empirical work that I do would be equally well served by linear regression and so I firmly fall into the camp of viewing it as a little bit pointless. Some disciplines with complex study designs (e.g. psychology) may find ANOVA useful.

[1] Gelman, A. (2005). Analysis of variance: why it is more important than ever (with discussion). Annals of Statistics 33, 1–53. doi:10.1214/009053604000001048

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thanks for the Gelman reference. I will read his paper. But, can't we analyze multilevel models using classical maximum likelihood? I agree that OLS is inefficient/inappropriate for multi-level models. $\endgroup$
    – user28
    Jul 23, 2010 at 15:50
  • 3
    $\begingroup$ @Srikant - there any many ways to deal with multilevel data and Gelman is "the king" of this field. His point is that ANOVA is a simple/clear method of capturing the key features of complex and hierarchical data structures or study designs and ANOVA is a simple/clear way of presenting the key results. In this sense it's role is complementary or exploratory. $\endgroup$
    – Graham Cookson
    Jul 23, 2010 at 16:30
  • 1
    $\begingroup$ +1 for a nice clear answer. Paragraph 3 is essentially what I was taught as a biology undergraduate, with emphasis laid on the ease of combining continuous and categorical independent variables in an ANOVA framework. $\endgroup$
    – Freya Harrison
    Mar 14, 2011 at 15:43
25
$\begingroup$

I think Graham's second paragraph gets at the heart of the matter. I suspect it's not so much technical than historical, probably due to the influence of "Statistical Methods for Research Workers", and the ease of teaching/applying the tool for non-statisticans in experimental analysis involving discrete factors, rather than delving into model building and associated tools. In statistics, ANOVA is usually taught as a special case of regression. (I think this is similar to why biostatistics is filled with a myriad of eponymous "tests" rather than emphasizing model building.)

Improve this answer
$\endgroup$
16
$\begingroup$

I would say that some of you are using the term regression when you should be using general linear model. I think of regression as a glm that involves continuous covariates. When continuous covariates are combined with dummy variables that should be called analysis of covariance. If only dummy variables are used we refer to that special form of glm as analysis of variance. I think analysis of variance has a distinct second meaning as the procedure for testing for significant coefficients in a glm using the decomposition of variance into model term components and the error term component.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ (+1) I have also immediately noted the ambiguous terminology "regression" throughout the discussion. $\endgroup$
    – Stéphane Laurent
    Aug 18, 2012 at 17:51
  • 1
    $\begingroup$ (+1) GLM might be the best way to disentangle different meanings. It should also be noted that in the history of ANOVA calculation procedures were used that obscure the relation betweeen OLS and ANOVA. The nomenclature might therefore be justified by historical reasons. $\endgroup$
    – jank
    Oct 24, 2013 at 17:35
10
$\begingroup$

ANOVA can be used with categorical explanatory variables (factors) that take more than 2 values (levels), and gives a basic test that the mean response is the same for every value. This avoids the regression problem on carrying multiple pairwise t-tests between those levels:

  • Multiple t-tests on a fixed 5% significance level, would make roughly 5% of them give wrong results.
  • These tests are not independed from each other. Comparing A's levels with B's is connected with comparing A's to C's, as A's data are used in both tests.

It is better to use contrasts for different combinations on the factor levels you want to test.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ You may wish to clarify this answer; as written, I see 3 issues. The first 2 are a little nit-picky but should still be edited, the third is substantive in the context of this discussion. (1) ANOVA can be used with only 2 groups (although most people just run a t-test then). (2) multiple t-tests w/ $\alpha=.05$ will asymptotically yield type I errors for 5% of those contrasts where no actual difference exists; how many errors will occur depends on how many nulls are true. $\endgroup$
    – gung - Reinstate Monica
    Dec 21, 2011 at 0:43
  • 8
    $\begingroup$ (3) your answer implies the problem of multiple comparisons applies to OLS regression, which it does not, when conducted properly. The proper way to test a factor in a regression context is to test the nested model with all factor dummies dropped against the full model with all factor dummies included. This test is identical to the one an ANOVA conducts. It is true that you should not use the tests of the individual dummy variables (which I suspect is what you are trying to describe here). $\endgroup$
    – gung - Reinstate Monica
    Dec 21, 2011 at 0:48
4
$\begingroup$

ANOVA you are testing whether there are significant difference between the population means assuming you are comparing more than two population means, then you are going to use an F test.

In regression analysis you build a model between independent variables and a dependent variable. If you have one independent variable with four levels you can use three dummy variables and run a regression model. The F-test for the regression model which is used to test for the significance of the regression model is the same as the F which you get when testing for the difference between the population means. If you run a stepwise regression then some of the dummy variables might be dropped from the model and your F-value will differ from that when you perform ANOVA test.

Improve this answer
$\endgroup$
1
  • 5
    $\begingroup$ This makes out ANOVA to be a testing procedure and regression to be a modelling procedure in which you can carry out tests. But ANOVA also has an underlying model, regardless of whether this is emphasised in all introductory treatments. So, this answer does not capture any difference between them. Nor is it addressed at the question, which is why they are taught as different regardless of strong similarities. $\endgroup$
    – Nick Cox
    Oct 20, 2013 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.