What is an F test?

Question

What is an F test and what does it show? Also what is an alpha, and how do I evaluate the p-value?

Answer 1

What is an F test is and what does it show?

The term "F test" may mean any test whose sampling distribution under the null hypothesis has an F-distribution. There are several quite distinct tests.

Here are a couple of the more common ones:

(i) an F-test for equality of means of multiple groups (also called ANOVA, short for ANalysis Of VAriance, since inequality of means will tend to lead to two different estimates of variance being different). If you reject the null, you have concluded that the sample means are too far apart to explain by random variation with equal population means; the alternative explanation is that the population means are not equal. For this test, large F values would cause you to reject the null hypothesis.

There are related tests, such as the overall F-test and partial-F tests for regression. The F-test in ANOVA can be seen as a special case of the regression one.

This test assumes normality but isn't especially sensitive to mild deviations from it.

(ii) a test for equality of variances. If you reject the null, you have concluded that the sample variances are too different to explain by random variation with equal variances; the alternative explanation is that the population variances are not equal.

This test** assumes normality and is quite sensitive to that assumption. I advise against using it, except under very limited circumstances. You are not likely to be in those circumstances.

** (Actually, there are a couple of these tests but I'm mostly referring to variance ratio test for equality of variances; Levene's test or the Brown-Forsythe test are used more nowadays and the strict assumption of normality is less of an issue)

There's additional discussion of both these tests (including some hopefully helpful diagrams) in this answer.

Also what is an alpha?

It is the chosen type I error rate$^\ddagger$. Before you test (indeed before you look at the data, and preferably before you even collect any data) you choose the chance you would reject null hypotheses that were true. It's also called the significance level.

It will probably help to read more about how hypothesis tests work in general. For example, see Wikipedia on the role of significance in hypothesis testing and type I and type II errors.

$^\ddagger$ more generally, it's the chosen upper bound on the type I error rate, since under composite null hypotheses you won't just have one.

... and how do I evaluate the p value?

It depends on whether you mean 'how do I calculate them?' or 'how do I interpret them?'

If you mean the first:

Calculation involves seeing "how extreme"$^\dagger$ the test statistic is.

$^\dagger$(deviant from the values of the test statistic you'd expect to see if the null hypothesis were true toward what you'd expect to see if it were false)

Calculation is done by computing the probability that a "random" test statistic from the null distribution would be at least as unusual as the calculated sample test statistic.

[The null distribution is the distribution the test statistic takes when the null hypothesis is true (an F-distribution for an F test), and more unusual here means further toward what you'd expect to see if the alternative hypothesis were true.]

So for an ANOVA, when the alternative is true (population means differ) the F-statistic tends to be larger, and you compute the probability that you'd see a value at least as large as the observed (sample) F if the null were true. That is, find the probability of being in the upper tail, at least as large as the observed value.

In practice one either looks up a table, or much more commonly nowadays, uses a computer program to find that tail probability.

If that's very small, then either the null hypothesis is true but a very low probability even occurred, or the null isn't true (and there's no need to think anything especially unlikely occurred). For sufficiently low p-values, the first position becomes untenable; you reject the null hypothesis. What makes it sufficiently low? Being smaller than the significance level, $\alpha$.

If you mean the second, it might help to read more about what a p-value is. We have many good posts on interpretation of p-values. Here are a few you might find helpful in some way:

How to interpret F- and p-value in ANOVA?

Is the exact value of a 'p-value' meaningless?

What is the meaning of a large p-value?

Misunderstanding a P-value?

What is the relationship between a p-value and a confidence interval?

The extent to which one can interpret a p-value (beyond it being above or below $\alpha$) is a matter of a bit of controversy. People who are more strictly in the Neyman-Pearson vein tend to say you shouldn't and people who take a more Fisherian approach (which distinction I haven't really covered) tend to be quite happy to do so. What we actually see in practice is usually a bit of a mixture of the two, a not entirely comfortable arrangement.