9
$\begingroup$

I want to learn ANOVA. Before I start to learn how the algorithm works (what calculations have to be done) and why it works, I first would like to know what problem do we actually solve with ANOVA, or what answer do we try to answer. In other words: What is input and what is output of the algorithm?

I do understand what we use as an input. We have a set of numbers. Each number comes with values of one or more categorical variables (also known as "factors"). For example:

+------------+------------+-------+
|   factor 1 |   factor 2 | value |
+------------+------------+-------+
|     "A"    |     "a"    |  1.0  |
|     "A"    |     "a"    |  2.4  |
|     "A"    |     "b"    |  0.3  |
|     "A"    |     "b"    |  7.4  |
|     "B"    |     "a"    |  1.2  |
|     "B"    |     "a"    |  8.4  |
|     "B"    |     "b"    |  0.4  |
|     "B"    |     "b"    |  7.2  |
+------------+------------+-------+

Is it correct to say that ANOVA calculates p-value of null hypothesis that states that there is no effect of the factors on the mean of the values? In other words, we give the above given data to the algorithm and as a result we get the p-value of the null hypothesis?

If it is the case, what measure do we actually use to calculate the p-value. For example we can say that, given the null hypothesis M can be as high as the observed one (or even higher) just by chance in 1% of cases. What is M?

Don't we also investigate factors in ANOVA separately? Can ANOVA say that factor_1 has an effect but factor_2 not? Can ANOVA say, that for a given factor values corresponding to is value "A", "B" and "C" are statistically indistinguishable (have the same mean, for example) but value "D" has an effect?

$\endgroup$
6
$\begingroup$

ANOVA stands for "Analysis of Variance". Rather unsurprisingly, it analyses variance.

Let's be a little more explicit. Your observations will exhibit some variance. If you group your observations by your factor 1, the variance within the groups defined by factor 1 will be smaller than the overall variance. Factor 1 "explains variance".

However, this is not sufficient to conclude that factor 1 actually does have a relationship to your observations... because grouping by anything whatsoever will "explain" variance. The good thing is that we know how much variance will be explained under the null hypothesis that your factor does, in fact, have nothing to do with your observations. This amount of variance explained under the null is described by an $F$ distribution.

Thus, the strategy in ANOVA is to estimate overall variance and within-groups variance (using sums of squares) and taking ratios of these estimated variances. This ratio is the $F$ statistic. We then compare this $F$ statistic to the critical value of the $F$ distribution in a one-sided test, yielding your $p$ value. The number of factor levels goes into one parameter of the $F$ distribution (more factor levels will explain more variance under the null hypothesis), and the number of observations and the number of levels goes into the other. This earlier question may be helpful.

(Why a one-sided test? Because, as above, any grouping will explain some variance, so it only makes sense to check whether your factor explains a significantly large amount of variance.)

The "Motivating Example" section of the Wikipedia entry provides some very nice illustrations of factors that explain very little, some, and a lot of the overall variance.

Two-way ANOVA and interactions, as in your example, as well as ANCOVA, are then just generalizations on this theme. In each case, we investigate whether adding some explanatory variable explains a significantly large amount of variance.

Once we have a significant overall $F$ test, we can examine whether certain factor levels' observations are significantly different than others in post-hoc tests. For instance, D may be different from A, B and C, but those may not be significantly different from each other. You will typically use $t$ tests for this. This earlier question may be useful, as well as this one.

$\endgroup$
  • $\begingroup$ So, we use all the number to calculate the overall variance $V$, then we calculate variances for each group $v_i$ and finally we combine all these variances (probably also with the group sizes) to get the "measure": $M = M (V, v_1, v_2, ..., v_k, n_1, n_2, ..., n_k)$. Then we calculate the probability of M to be as big as it is or even larger under assumption that the null hypothesis is correct. $\endgroup$ – Roman Nov 25 '15 at 16:35
  • $\begingroup$ Exactly. $M$ is your $F$ statistic. Here is the actual formula. $\endgroup$ – Stephan Kolassa Nov 25 '15 at 16:38
  • $\begingroup$ To be honest I am still a bit confused. As far as I got you, ANOVA returns the p-value of the null-hypothesis. But on the other hand, from the "Motivating Example" from Wikipedia one can conclude that ANOVA gives us the best factor (or a combination of factors), that "explains" the data the best. So, in the example ANOVA says that breed is the best factor to explain the weight of the dogs. $\endgroup$ – Roman Nov 25 '15 at 16:55
  • 1
    $\begingroup$ "Best" is loaded. This strays into stepwise-model-selection-based-on-p-values territory, and that is problematic. Don't read too much into the motivating example. The best thing about it is the depiction of variance explained (zero, a little, a lot). Better to go down and read about how the $F$ statistic is calculated based on sums of squares, and remember that those sums of squares are just estimators of variances. $\endgroup$ – Stephan Kolassa Nov 25 '15 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.