Can you explain ANOVA and its assumptions to a beginner?

Question

I was about to start coding up an ANOVA test to study the differences in house prices between neighborhoods. I read that ANOVA is a great way to find out if there is significance difference in nominal groups of data against a continuous variable.

But then I read about the three assumptions that have to be fulfilled in order to make sure that your ANOVA results can be trusted:

1. The experimental errors of your data are normally distributed
2. Equal variances between treatments - Homogeneity of variances, Homoscedasticity
3. Independence of samples - Each sample is randomly selected and independent

So, part 1 leads me to believe that you must first make some predictions based on your data, and then check the errors.

Does that mean ANOVA tests are always done post-hoc? Or is it talking about the difference between a sample and the mean?

I ask, because I am looking for a way to forecast whether or not a given nominal variable has any significance to a linear regression model. If ANOVA must be done AFTER modeling, then I might as well run the model with and without a given variable, and see which performs better.

Next, I read that you have to perform statistics tests on each of your three assumptions. So basically, to a novice like me, it looks like you are running tests on tests on tests.

In application, how often are these assumptions tested? How often do such tests fail?

I'm just a little blown away how complicated this stuff seems at first look. My stats background is pretty much zilch so forgive my lack refinement.

In the second assumption, what is meant by the word "treatment"

Again, I could just use the formulas on Wikipedia to code something up, but I don't want to be a lazy analyst by glossing over possibly important details! This is what I get for not taking any stats as a math major!

Answer 1

I think this is a great question. First of all, I want to warn you that there are often significant differences between statistics as presented in textbooks and statistics used in practice. So even though you read in the textbook that you need to do this and that and everything before doing an ANOVA, in practice this is hardly the case.

In practice ANOVA is a very simple test for a very simple problem. It seems to me from your post that you may have come from a machine learning background where the modeling is much much more sophisticated than ANOVA. ANOVA dates back to the early half of the last century where statistical tests were still calculated by hand. At that time, it was a clever trick to test for the equality of means between different groups. It has more sophisticated variants, e.g. two-way, three-way ANOVA, ANCOVA, or even MANOVA. But all of these were designed to be done without computers, and in fact, all of them could be done equivalently using some kind of linear regression.

In answering your questions:

1. Does that mean ANOVA tests are always done post-hoc? Or is it talking about the difference between a sample and the mean?

Yes and no. In fact, what you mean by "post-hoc" in your question is not what statisticians generally regard as "post-hoc". "post-hoc" in traditional statistics means carrying hypothesis tests that are designed after they have examined the data. For example, if you had decided to test whether house prices were different in different neighbourhoods only after you've examined the data graphically, then it would be "post-hoc". If you had wanted to do this before seeing the data, and then you want to look for a test to do this properly, that's not "post-hoc".

Secondly, ANOVA is concerned only with the comparison of the means of the groups. So, it is not necessary to do any more complicated modelling. "Error" simply means difference from the mean. There is a variant of ANOVA, called ANCOVA, that deals with the case where you have other "covariates" you want to adjust for, but you might as well just use linear regression in that case.

Thirdly, performing tests to determine whether the assumptions are met is sometimes recommended in textbooks, but this is in fact not always advisable. First of all, it may be a matter of "who cares", because often tests like these are meant to be exploratory anyway, i.e., to give the data analyst a better grasp of the structure of the data. Secondly, data analysis in practice is not applying an algorithm. It's not the case of "if stage 1 is significant then do test A, if not then test B". Far more often the checks are done graphically or informally to make sure that the assumptions are not too far off.

2. In application, how often are these assumptions tested? How often do such tests fail?

So I guess the above answers your second question too.

3. In the second assumption, what is meant by the word "treatment"

In textbook presentations of ANOVA, the scenario they have in mind is often the determination of whether a "treatment" (of a disease, say) is better than a "control". The hypothesis is that if it is better, then the means of the two groups (treatment vs. control) would be different. Sometimes they have more than two groups (more than 1 treatment) and thus they would use ANOVA.

BTW, if your aim is to study house prices (rather than to study statistics), then there are likely better methods in your case than ANOVA, especially if your data is bigger than the toy examples you see in textbooks.