What is an F test and what does it show? Also what is an alpha, and how do I evaluate the p-value?

  • 1
    $\begingroup$ have you read the corresponding wiki? $\endgroup$ – user603 Dec 7 '13 at 22:59
  • 2
    $\begingroup$ Please give more specific details as to where you are confused after reading some relevant material through an internet search. Show what you've tried to do to solve the problem yourself. $\endgroup$ – John Dec 7 '13 at 23:07
  • $\begingroup$ Yes. However my confusion lies in the fact that I am trying to test for a difference between two large data sets, one for males and if they like chocolate or not, and one for females, and if they like chocolate or not. I have tried transformations of the data, which have not improved the data to make it fit a more normal distribution. In this case, I do not know whether to use a parametric or non parametric test, and I do not really understand what this test result is telling me in accordance to my data set. $\endgroup$ – user3069564 Dec 7 '13 at 23:08
  • 4
    $\begingroup$ If this is for homework or a class assignment please add the self-study tag. If it is for anything else, I suggest hiring a statistician. $\endgroup$ – Peter Flom Dec 8 '13 at 0:05
  • 2
    $\begingroup$ Hang on ... you have a statistics assignment without having done any stats? How does a responsible university let you do that? $\endgroup$ – Glen_b Dec 8 '13 at 0:31

What is an F test is and what does it show?

The term "F test" may mean any test whose sampling distribution under the null hypothesis has an F-distribution. There are several quite distinct tests.

Here are a couple of the more common ones:

(i) an F-test for equality of means of multiple groups (also called ANOVA, short for ANalysis Of VAriance, since inequality of means will tend to lead to two different estimates of variance being different). If you reject the null, you have concluded that the sample means are too far apart to explain by random variation with equal population means; the alternative explanation is that the population means are not equal. For this test, large F values would cause you to reject the null hypothesis.

There are related tests, such as the overall F-test and partial-F tests for regression. The F-test in ANOVA can be seen as a special case of the regression one.

This test assumes normality but isn't especially sensitive to mild deviations from it.

(ii) a test for equality of variances. If you reject the null, you have concluded that the sample variances are too different to explain by random variation with equal variances; the alternative explanation is that the population variances are not equal.

This test** assumes normality and is quite sensitive to that assumption. I advise against using it, except under very limited circumstances. You are not likely to be in those circumstances.

** (Actually, there are a couple of these tests but I'm mostly referring to variance ratio test for equality of variances; Levene's test or the Brown-Forsythe test are used more nowadays and the strict assumption of normality is less of an issue)

There's additional discussion of both these tests (including some hopefully helpful diagrams) in this answer.

Also what is an alpha?

It is the chosen type I error rate$^\ddagger$. Before you test (indeed before you look at the data, and preferably before you even collect any data) you choose the chance you would reject null hypotheses that were true. It's also called the significance level.

It will probably help to read more about how hypothesis tests work in general. For example, see Wikipedia on the role of significance in hypothesis testing and type I and type II errors.

$^\ddagger$ more generally, it's the chosen upper bound on the type I error rate, since under composite null hypotheses you won't just have one.

... and how do I evaluate the p value?

It depends on whether you mean 'how do I calculate them?' or 'how do I interpret them?'

If you mean the first:

Calculation involves seeing "how extreme"$^\dagger$ the test statistic is.

$^\dagger$(deviant from the values of the test statistic you'd expect to see if the null hypothesis were true toward what you'd expect to see if it were false)

Calculation is done by computing the probability that a "random" test statistic from the null distribution would be at least as unusual as the calculated sample test statistic.

[The null distribution is the distribution the test statistic takes when the null hypothesis is true (an F-distribution for an F test), and more unusual here means further toward what you'd expect to see if the alternative hypothesis were true.]

So for an ANOVA, when the alternative is true (population means differ) the F-statistic tends to be larger, and you compute the probability that you'd see a value at least as large as the observed (sample) F if the null were true. That is, find the probability of being in the upper tail, at least as large as the observed value.

In practice one either looks up a table, or much more commonly nowadays, uses a computer program to find that tail probability.

If that's very small, then either the null hypothesis is true but a very low probability even occurred, or the null isn't true (and there's no need to think anything especially unlikely occurred). For sufficiently low p-values, the first position becomes untenable; you reject the null hypothesis. What makes it sufficiently low? Being smaller than the significance level, $\alpha$.

If you mean the second, it might help to read more about what a p-value is. We have many good posts on interpretation of p-values. Here are a few you might find helpful in some way:

How to interpret F- and p-value in ANOVA?

Is the exact value of a 'p-value' meaningless?

What is the meaning of a large p-value?

Misunderstanding a P-value?

What is the relationship between a p-value and a confidence interval?

The extent to which one can interpret a p-value (beyond it being above or below $\alpha$) is a matter of a bit of controversy. People who are more strictly in the Neyman-Pearson vein tend to say you shouldn't and people who take a more Fisherian approach (which distinction I haven't really covered) tend to be quite happy to do so. What we actually see in practice is usually a bit of a mixture of the two, a not entirely comfortable arrangement.

  • $\begingroup$ Thank you very much Glen_b for this answer. It is very kind of you and helps me alot. :-) $\endgroup$ – user3069564 Dec 8 '13 at 14:02
  • $\begingroup$ If you are able to answer some of the questions in my answer that request clarification, I may be able to add some detail. $\endgroup$ – Glen_b Dec 8 '13 at 15:04
  • 1
    $\begingroup$ Thanks for the additions, I think they have greatly improved the answer. It seems quite difficult to me to write an answer that is non-technical in nature, by dint of the nature of the question. I think I might be tempted to expand on the regression aspect a little (if only to explain what hypotheses it's testing, which are quite different to "equality of means" in the bread-and-butter ANOVA) since a lot of people will encounter F for the first time in that context, but the answer is already pretty long. $\endgroup$ – Silverfish Aug 12 '15 at 23:23
  • 1
    $\begingroup$ @Silverfish Actually, the regression test is testing an "equality of means" null hypothesis against an alternative that the conditional means of $Y$'s differ (in the manner specified according to the linear model). I'll think about expanding on it, but I hesitate because of the length. $\endgroup$ – Glen_b Aug 13 '15 at 1:57
  • 1
    $\begingroup$ That's a fair point re equality of means under the null model. Pitched at the level of the kind of reader who is going to be searching for this kind of question, I'd be prepared to wager their lecturer has expressed the hypotheses in terms of slopes/coefficients being zero, so the equivalence may not be so obvious to them (particularly if they have never stopped to think what the null model looks like graphically). It was just a minor point, however, and this is a very healthy-looking answer. $\endgroup$ – Silverfish Aug 13 '15 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.