Reasoning for comparing variance between and within for Anova?

Question

I looking for some logic reasoning why ANOVA test uses $σ^2$ given by these to variables. a logical reasoning why the ratio of variance between these two would say that the each population has the same µ, and the groups which it has been tested on comes from the same overall distribution Why??

Answer 1

The actual logic behind it is much simpler:

There are two ways to estimate the variance in the population when you have several groups but you assume the values all come from the same distribution.

1. look at the variance in each group

Each group can be considered a random sample from the population and each one provides an estimate for the population variance. We need a single estimate, so we take the average. This is the within-variance $\hat{\sigma}_\mathrm{within}^2$

2. look at the variation of the means

The means of the different groups can be seen as a sampling distribution of means from the population. The variance of this distribution is: $\hat{\sigma}^2_{\bar{x}} = \frac{\hat{\sigma}^2}{n}$

Crucially, we are interested in the population variance $\hat{\sigma}^2$ so we have to multiply the variance of the means with the sample size: $\hat{\sigma}^2 = \hat{\sigma}^2_\bar{x} n = \hat\sigma^2_\mathrm{between}$

If all values really come from a single distribution, the two ways to estimate the variance should give the same result. But obviously drawing samples is a random process, so we calculate a $p$-value, which tells us the probability of deviations from equal variances (even though all values come from a single population). If the probability is very low, we might argue that the values do not come from a single population. Then the means will variate much more than we expected and the ratio increases.

To me its astounding that ANOVA is not taught in this way. Most instructors just start with sum of squares and degrees of freedom and the formulas do not make any sense. When seeing that both variances are actually estimates of the population variance, the formulas start to make sense.

Note that I assumed a balanced design where the sample size for each group is the same ($n$). If this is not the case one has to take weighted averages.

Answer 2

If I understand correctly, I can begin my answer by saying that we assume that the sample means differ; if their observed differences were 0, there would be nothing to "test," so to speak. Now what we are actually interested in is if these observed mean differences are merely due to chance (i.e., sampling error) or represent a real effect of the independent variable (i.e., factor) on the dependent variable in the real world. That is, we want to ensure that these observed differences are reliable so that we can be comfortable making an inference to the population (as is the case in inferential statistics in general).

To do so, we partition the variation seen across the observed scores into two sources: between-group variance, that is attributable to the factor (i.e., the independent variable), and within-group variance, whatever is left over and considered random error. The logic is that the latter (i.e., the denominator) is a representation of the variation we would expect simply by chance. The value of this ratio is distributed as F when the null hypothesis of no difference between population means is true.

However, when the numerator is large enough to yield an F-value that is also sufficiently large (typically reaching the criterion of p < .05), then it is often concluded that the observed difference between means is not merely due to chance (sampling error). This simply rests on the meaning of the derived p-value. Of course, remember that the ANOVA significance test only tests if one mean is different from any other; to get at which means differ in particular, one needs to run post-hoc tests. However, to my understanding (and anyone feel free to correct me if I'm wrong), but partitioning the observed variance in this way is all about getting obtaining an F-statistic and its associated p-value. In other words, it is all about hypothesis testing, not estimation (which is done by calculating the raw difference between means and possibly a confidence interval around it--at least in the frequentist approach).

Answer 3

I'll start by discussing what happens under the null hypothesis of no difference between groups. Suppose we run an ANOVA on junk data. Let's say the groups are the ethnic/racial divisions as devised by the US Census and the variable is a random number generated by a computer. In our sample, each group will have a different mean value of RN (at least, if we use enough decimal places) so there will, even under the null, be some variation between groups. There will also be variation within groups since, e.g. not all White people have the same RN (indeed, no two people do). In this case, we expect the variation within and between to be equal - it's all noise - so we expect an F = 1.

Now, what happens when the null is false? Suppose we no longer use random numbers but family income. There will still be variation within and between, but we now expect that the variation between groups will be larger than the variation within, because the incomes are not distributed randomly, but with some order.