3
$\begingroup$

I was about to start coding up an ANOVA test to study the differences in house prices between neighborhoods. I read that ANOVA is a great way to find out if there is significance difference in nominal groups of data against a continuous variable.

But then I read about the three assumptions that have to be fulfilled in order to make sure that your ANOVA results can be trusted:

  1. The experimental errors of your data are normally distributed
  2. Equal variances between treatments - Homogeneity of variances, Homoscedasticity
  3. Independence of samples - Each sample is randomly selected and independent

So, part 1 leads me to believe that you must first make some predictions based on your data, and then check the errors.

Does that mean ANOVA tests are always done post-hoc? Or is it talking about the difference between a sample and the mean?

I ask, because I am looking for a way to forecast whether or not a given nominal variable has any significance to a linear regression model. If ANOVA must be done AFTER modeling, then I might as well run the model with and without a given variable, and see which performs better.

Next, I read that you have to perform statistics tests on each of your three assumptions. So basically, to a novice like me, it looks like you are running tests on tests on tests.

In application, how often are these assumptions tested? How often do such tests fail?

I'm just a little blown away how complicated this stuff seems at first look. My stats background is pretty much zilch so forgive my lack refinement.

In the second assumption, what is meant by the word "treatment"

Again, I could just use the formulas on Wikipedia to code something up, but I don't want to be a lazy analyst by glossing over possibly important details! This is what I get for not taking any stats as a math major!

$\endgroup$
5
  • 2
    $\begingroup$ This is a big ask. There are books on anova. Perhaps you could try to break down your questions to make them smaller. $\endgroup$ May 5, 2020 at 2:09
  • $\begingroup$ @JeremyMiles, I'm looking for a quick and dirty to speed up application. Do you suggest that each of my three questions be posted separately? Or do you suggest breaking it down even further? I will admit, there is a lot I don't understand. I've been reading for the past 4 hours, and not gaining much traction. $\endgroup$ May 5, 2020 at 2:12
  • $\begingroup$ See stats.stackexchange.com/questions/40549/… $\endgroup$ May 5, 2020 at 2:22
  • $\begingroup$ @kjetilbhalvorsen That's a great intuitive explanation of ANOVA. Unfortunately, I get the surface level idea of what's going on, but I'm looking for more nitty gritty details. Check out the specific questions in my post. $\endgroup$ May 5, 2020 at 2:40
  • 3
    $\begingroup$ Yeah, I would ask three. And try to make them specific - there are plenty of answers about (say) normality already on CrossValidated. Edit: But there you go, an answer. Shows what I know. :) $\endgroup$ May 5, 2020 at 3:15

1 Answer 1

3
$\begingroup$

I think this is a great question. First of all, I want to warn you that there are often significant differences between statistics as presented in textbooks and statistics used in practice. So even though you read in the textbook that you need to do this and that and everything before doing an ANOVA, in practice this is hardly the case.

In practice ANOVA is a very simple test for a very simple problem. It seems to me from your post that you may have come from a machine learning background where the modeling is much much more sophisticated than ANOVA. ANOVA dates back to the early half of the last century where statistical tests were still calculated by hand. At that time, it was a clever trick to test for the equality of means between different groups. It has more sophisticated variants, e.g. two-way, three-way ANOVA, ANCOVA, or even MANOVA. But all of these were designed to be done without computers, and in fact, all of them could be done equivalently using some kind of linear regression.

In answering your questions:

  1. Does that mean ANOVA tests are always done post-hoc? Or is it talking about the difference between a sample and the mean?

Yes and no. In fact, what you mean by "post-hoc" in your question is not what statisticians generally regard as "post-hoc". "post-hoc" in traditional statistics means carrying hypothesis tests that are designed after they have examined the data. For example, if you had decided to test whether house prices were different in different neighbourhoods only after you've examined the data graphically, then it would be "post-hoc". If you had wanted to do this before seeing the data, and then you want to look for a test to do this properly, that's not "post-hoc".

Secondly, ANOVA is concerned only with the comparison of the means of the groups. So, it is not necessary to do any more complicated modelling. "Error" simply means difference from the mean. There is a variant of ANOVA, called ANCOVA, that deals with the case where you have other "covariates" you want to adjust for, but you might as well just use linear regression in that case.

Thirdly, performing tests to determine whether the assumptions are met is sometimes recommended in textbooks, but this is in fact not always advisable. First of all, it may be a matter of "who cares", because often tests like these are meant to be exploratory anyway, i.e., to give the data analyst a better grasp of the structure of the data. Secondly, data analysis in practice is not applying an algorithm. It's not the case of "if stage 1 is significant then do test A, if not then test B". Far more often the checks are done graphically or informally to make sure that the assumptions are not too far off.

  1. In application, how often are these assumptions tested? How often do such tests fail?

So I guess the above answers your second question too.

  1. In the second assumption, what is meant by the word "treatment"

In textbook presentations of ANOVA, the scenario they have in mind is often the determination of whether a "treatment" (of a disease, say) is better than a "control". The hypothesis is that if it is better, then the means of the two groups (treatment vs. control) would be different. Sometimes they have more than two groups (more than 1 treatment) and thus they would use ANOVA.

BTW, if your aim is to study house prices (rather than to study statistics), then there are likely better methods in your case than ANOVA, especially if your data is bigger than the toy examples you see in textbooks.

$\endgroup$
5
  • $\begingroup$ BTW, there is usually no need to "code up" an ANOVA, because it is readily available in any statistical software, unless you want to study how it's done. $\endgroup$
    – Tim Mak
    May 5, 2020 at 3:10
  • $\begingroup$ Thanks! This is a great answer to my question. You are correct about where my questions come from. Essentially, I am looking to do two things: 1.) Get insight into how important neighborhood is to sale price. I care about understanding my data, not just for my machine learning, but also for my education. 2.) I'm looking for an answer to drop/keep when it comes to data I am feeding to a regression model. $\endgroup$ May 5, 2020 at 3:17
  • $\begingroup$ Before I accept, do you mind if I ask a clarifying question? Namely, I have looked at my data. I've looked at boxplots, ordered by median. I can also visually inspect the mean sale price of homes in each neighborhood and see that they are different one from another. Are you saying that this visual check is enough to make ANOVA pretty useless to me? Another person has told me that you cannot trust this sort of visual analysis and that it's imperative to dig deeper. What say you? $\endgroup$ May 5, 2020 at 3:22
  • 1
    $\begingroup$ My point is. Did you come up with the question itself after seeing the data? If not, then it's not post-hoc. Even if you did, it's generally no problem as long as you make it clear to the readers of your report. $\endgroup$
    – Tim Mak
    May 5, 2020 at 3:24
  • $\begingroup$ Note that there is a scenario where data analysis can be like applying an algorithm, namely in clinical trials, where the statistical analysis procedure has to be specified beforehand before the trial begins in as much details as possible. This is to ensure strict control of Type 1 error. $\endgroup$
    – Tim Mak
    May 5, 2020 at 3:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.