3
$\begingroup$

I am wondering about the normality assumption. It is required because underneat all this, ANOVA is running a F test. The denominator of the F test is the sd of the residuals, thus we need these residuals to be normally distributed.

This leaves me wondering about the numerator: the numerator is the variance of means of the factors (the k samples we are testing).

I have read that if the residuals are normally distributed, then, under the null (all population means are equal), the observed means also are, but I can not find any justification.

In fact, in the case where the samples have unequal sizes, and unequal variances, these k means belong to k different sampling distributions of the means, so they do not appear to be even i.d.

Could someone explain how we ensure that the numerator of the F statistic is also X2 distributed under the null?

Or do we just ignore the numerator, because it (usually, but not always) has such a small sample size (k is often small)?

I do know that the F test is robust to departures from normality, but... I am interested in the rigorous assumptions which would need to be satisfied to run a "mathematically" valid F test. How is normality of the k means ensured?

$\endgroup$
1
  • $\begingroup$ I removed my answer. It wasn't clear enough and should be made more in line with your comments. I didn't realize that you in particular was interested in the deviations form the usual assumptions and thought that you did not understand why in the ideal situation the normality leads to chi squares. I will try again later if no one else already gave a better answer than mine. $\endgroup$
    – BenP
    Commented Apr 2 at 9:28

2 Answers 2

1
$\begingroup$

This is straightforward when the groups have the same size, $n$.

The group sample means, under the null hypothesis that the population means all equal some $\mu$ and with the groups assumed to be Gaussian with equal variances, are distributed the usual way, $N\left(\mu, \frac{\sigma^2}{n}\right)$. This gives us the desired Gaussian distribution.

Perhaps this can inspire some thinking for handling the added wrinkle of $n$ not being the same for each group. The group sample means in that case still have Gaussian distributions, but those Gaussian distributions are not all the same (same mean but different variance).

$\endgroup$
0
$\begingroup$

Let's assume there are 3 groups with different group sizes $n_1$, $n_2$ and $n_3$. The observations in the groups are normally distributed: $N(\mu_1,\sigma)$, $N(\mu_2,\sigma)$ and $N(\mu_3,\sigma)$. So it is assumed that the within-group standard deviations are equal.

The group means $\bar{X_1}$, $\bar{X_2}$ and $\bar{X_3}$ must then also normally distributed: $N(\mu_1,\frac{\sigma}{\sqrt{n_1}})$, $N(\mu_2,\frac{\sigma}{\sqrt{n_2}})$ and $N(\mu_3,\frac{\sigma}{\sqrt{n_3}})$. So, the normality assumption of the group means, used to calculate the numerator of the F statistic, is based on the fact that the original observations of each group are drawn from a normal population (but even if it would not be perfectly normal, the central limit theorem "guarantees" that it is reasonable to assume that each group's mean comes form a normal distribution, given some "large enough" group size).

Regarding the $\chi^2$ distribution, the following may be helpful. In general, a $\chi^2$ distribution with say 3 df, is equal to the sum of three independent and squared standardized normal distributions. If we standardize the group means, take the squares and sum we get:

$(\frac{\bar{X_1}-\mu_1}{\sigma/\sqrt{n_1}})^2 + (\frac{\bar{X_2}-\mu_2}{\sigma/\sqrt{n_2}})^2 + (\frac{\bar{X_3}-\mu_3}{\sigma/\sqrt{n_3}})^2$

and this sum has a $\chi^2$ with 3 df. But now suppose the null hypothesis is true, then the 3 group means would all be equal to the overall mean $\mu$ and we would have:

$(\frac{\bar{X_1}-\mu}{\sigma/\sqrt{n_1}})^2 + (\frac{\bar{X_2}-\mu}{\sigma/\sqrt{n_2}})^2 + (\frac{\bar{X_3}-\mu}{\sigma/\sqrt{n_3}})^2$

which again would have a $\chi^2$ distribution with 3 df. However, if we replace the unknown $\mu$ with its sample estimate $\bar{X}$, we get:

$(\frac{\bar{X_1}-\bar{X}}{\sigma/\sqrt{n_1}})^2 + (\frac{\bar{X_2}-\bar{X}}{\sigma/\sqrt{n_2}})^2 + (\frac{\bar{X_3}-\bar{X}}{\sigma/\sqrt{n_3}})^2$

which can be proven to follow a $\chi^2$ distribution with 3-1=2 df. This last sum of squares can also be written as:

$\frac{n_1(\bar{X_1}-\bar{X})^2}{\sigma^2} + \frac{n_2(\bar{X_2}-\bar{X})^2}{\sigma^2} + \frac{n_3(\bar{X_3}-\bar{X})^2}{\sigma^2} = \frac{SSB}{\sigma^2}$

So the sum of squares between or $SSB$ divided by $\sigma^2$ follows a $\chi^2$ distribution with 2 df. That is, assuming that the original observations are normally distributed leads to the group means being normally distributed, which finally leads to the result $\frac{SSB}{\sigma^2}$ is distributed as $\chi^2$, which is needed for the $F$ distribution to hold.

Further, for sum of squares WITHIN, or $SSW$ we know (you did not mention that as a problem) that $\frac{SSW}{\sigma^2}$ follows a $\chi^2$ distribution with $N-3$ df, where $N=n_1+n_2+n_3$ is the total number of observations in the three groups. For the $F$ we divide both $\chi^2$'s by their df's, so we get:

$F = \frac{SSB/(r-1)}{\sigma^2} / \frac{SSW/(N-3)}{\sigma^2} = MSB / MSW$

where MSB and MSW denote the mean squares between and within.

And then there is the question of the error terms being normally distributed versus the X values themselves. If the null hypothesis is true, all errors $\epsilon$ can be notated as $\epsilon = X-\mu$. This means that if the form of the distribution of the error terms is normal, this also holds for the original X values, and vice versa. Both distributions are assumed to be normal with the same standard deviation $\sigma$, only their means differ. If the null hypothesis is NOT true, the errors in each group are equal to the observed X values minus the group mean, as in $\epsilon = X - \mu_1$ for group 1. Within each group, the errors then have the same normal distribution as the original $X$ values in the group. Obviously, the distribution of all X values in the three groups taken together, is not normal anymore! The assumption of normality is about the error terms or about the X values WITHIN each of groups.

Finally one can have the problem of heteroscedasticity or different variances in the groups used in the anova. As you say, the Welch approach seems to do well in that case. I never studied it, so I cannot help you further with that. Good question!

$\endgroup$
5
  • $\begingroup$ You seem to have 3 assumptions: normality of the groups, equal sample sizes, and equal variances. But, equal sample sizes is absolutely not a requirement to run a 1-way ANOVA, nor is equality of variances (there is a Welch "flavor", which adjusts the d.f.'s, which works so well that several sources suggest to only use that flavor) So as I posted in the question, in the case of unequal sample sizes, and unequal variances, how can one state that the numerator of the F is χ2 ? $\endgroup$
    – jginestet
    Commented Apr 2 at 2:19
  • $\begingroup$ Also, normality of the groups does NOT seem to be a necessary assumption for 1-way ANOVA, but instead normality of the residuals (which then ensures that the denominator of the F is a χ2 . See stats.stackexchange.com/questions/437414/…, and stats.stackexchange.com/questions/6350/… So that thickens the plot even more; how can the numerator be a χ2 (is a sum of squared Z's)? $\endgroup$
    – jginestet
    Commented Apr 2 at 2:23
  • $\begingroup$ @jginestet I will change my answer, as it could be made more clear. Also I will add a few things. $\endgroup$
    – BenP
    Commented Apr 2 at 8:28
  • $\begingroup$ The mention of the central limit theorem gets at the robustness of ANOVA that is important for real empirical work but separate from the question, which concerns the statistical theory under a strict Gaussian assumption. $\endgroup$
    – Dave
    Commented Apr 2 at 11:00
  • $\begingroup$ Good point @Dave, I will put in in parentheses. $\endgroup$
    – BenP
    Commented Apr 2 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.