Why doesn’t a $F$-statistic of 1 result in a $p$-value of about 0.5?

Question

I’ve been presented with the following chain of reasoning.

1. In an ANOVA, if the null hypothesis is true the F-statistic is expected to be about 1.

2. A p-value is the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct.

3. Therefore, a F-statistic of 1 should result in a p-value of about 0.5.

I sense that this can’t be correct in general, because playing around in R it seems like the p-value only comes out as 0.5 when $d_1 = d_2$.

d1 <- 1  
d2 <- 200  
1-pf(1, d1, d2)

However, I don’t know how to explain why it is not in general true that an F-statistic of 1 results in a p-value of about 0.5, but why it is true when $d_1 = d_2$.

Answer 1

I think this chain of reasoning contains a bit of confusion between mean and median. A p-value of 0.5 should be expected at the theoretical median (i.e. $F^{-1}(0.5)$), not the expected value. From looking at the CDF of a F-distribution, it seems that the median is at 1 for $d_1 = d_2$, but not for $d_1 \neq d_2$, which is in line with your observations.

Answer 2

Suppose a one-way ANOVA has three levels of the factor, and that the three levels are from exactly the same normal distribution. So if each level has 20 replications, the data for the three levels can be sampled (simulated) as shown below. I'll use oneway.test to do the ANOVA because it is simplest to use.

set.seed(906)
x = rnorm(3*20, 100, 15);  g = rep(1:3, each=20)
oneway.test(x ~ g)

        One-way analysis of means (not assuming equal variances)

data:  x and g
F = 0.77799, num df = 2.000, denom df = 37.441, p-value = 0.4666

For these particular data $F = 0.78 \ne 1,$ and the P-value, about $0.47,$ far too large for the null hypothesis to be rejected. (Failing to reject is the correct decision because all three groups are from exactly the same distribution.) Also, the P-value is not far from your speculated value $0.5.$

Now let's try $100\,000$ such ANOVAs and look at the overall behavior of the P-value.

set.seed(906)
pv = replicate(10^5, oneway.test(rnorm(60,100,15)~g)$p.val)
summary(pv)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
0.0000102 0.2520117 0.4995210 0.5004963 0.7501248 0.9999940 

The mean P-value is about $0.5$ as you suggested. But more precisely, the distribution of the $100\,000$ P-values is nearly standard uniform.

hist(pv, prob=T, col="skyblue2")
curve(dunif(x, 0,1), add=T, col="red", n=10001, lwd=2)

Note: To be absolutely honest, maybe the distribution of the P-value here not exactly standard uniform here. The procedure oneway.test uses an approximate test statistic.

This ANOVA does not require group variances to be equal and the F-statistic is slightly altered to allow for differences among group sample variances. The alteration is relatively small here because the groups have equal population variances.

Answer 3

There is more than one type of ANOVA test, the one I will use to try and provide insight about this question is the F-test applied to linear regression where the null hypothesis is that all of the regressors equal zero.

1. Roughly this is correct, but the value of the F-statistic gets bigger with more regressors in a model. Let us say you take a regression model and add another regressor to it. Then this will fit at least as well as the original model. The numerator of the newer model will likely be greater than before and the denominator likely smaller. Thus the calculated F-statistic from your new model will be greater than or equal to the original.

2. Yes. Although its slightly worse than that as this all depends on the assumptions required of the particular test being correct. A low p-value tells you something about a model is unlikely, and it is hoped that it is the null hypothesis part rather than the assumptions of the particular statistical test part.

3. This is not taking into account part 1., that the value of the F-statistic obtained depends on the degrees of freedom. The degrees of freedom are affecting the cumulative distribution in the ways you have discovered by simulation.

Answer 4

Point ‘1’ is basically OK. If the ANOVA null hypothesis is true, the expected value of $p$ is $0.5$, and @BruceET’s answer helps to build intuitions about why that is the case. “About $1$” is a reasonable gloss for the expected value of $F$ under the ANOVA null hypothesis, although how close to $1$ depends on the value of $d_2$. More precisely, the expected value of $F$ under the ANOVA null hypothesis is $\frac{d_2}{d_2−2}$).

Point ‘2’ is fine.

The real problem occurs in point ‘3’. As @nope notes, a $p$-value of $0.5$ should be expected at the theoretical median ($F^{−1}(0.5)$), and not the expected value of, $F$.

I present some further discussion, which will be overly basic for some viewers of this site, but which was helpful for convincing my interlocutor that something had indeed gone wrong at point ‘3’.

In ANOVA applications $d_1$ will be $< d_2$, seeing as $d_1$ is calculated as $k-1$, while $d_2$ is calculated as $N-k$, where $N$ is the sample size and $k$ is the number of groups.

While $d_1 < d_2$ the actual distribution of $F$s under the null hypothesis contains many $F$-values $< 1$, with the average $F$-value dragged upwards to $≈1$ by the occasional large $F$-value. Thus the median $F$ produced under the null hypothesis (the $F$ which accords to $p=0.5$) is $< 1$.

I have pasted below some R code which generates a plot of the empirical distribution of randomly sampled $F$-values in an ANOVA scenario when there are $3$ groups of $30$ subjects (i.e. $d_1=2,d_2=87)$, and the null hypothesis is true.

It’s easy to see that this closely matches the theoretical distribution of $F$-values.

number_of_groups <- 3
group_size <- 30
mean <- 100
sd <- 15
num_samples <- 30000
percentile <- 50 # 50 for median, 95 for critical F-value at α=0.05, etc

sampled_Fs <- vector(mode = "numeric", length = num_samples) 
sampled_Ps <- vector(mode = "numeric", length = num_samples) 

d1 <- number_of_groups - 1
d2 <- group_size * number_of_groups - number_of_groups

for(i in 1:num_samples) {
  
  x = rnorm(number_of_groups*group_size, mean, sd)
  g = rep(1:number_of_groups, each=group_size)
  
  ANOVA_results <- aov(x ~ as.factor(g))
  sampled_Fs[i] <- summary(ANOVA_results)[[1]][["F value"]][[1]]
  sampled_Ps[i] <- summary(ANOVA_results)[[1]][["Pr(>F)"]][[1]]
  
} 


sprintf("Under the null hypothesis the expected value of F(d1=%d,d2=%d) is %f", d1, d2, (d2/(d2-2)))

sprintf("Across %d random samples, the mean F(d1=%d,d2=%d) was %f", num_samples, d1, d2, mean(sampled_Fs))

sprintf("Across %d random samples, the mean p-value was %f", num_samples, mean(sampled_Ps))

sprintf("Under the null hypothesis the %fth percentile of the F-value (d1=%d, d2=%d) is expected to be %f", percentile, d1, d2, qf(percentile/100,d1,d2))

sprintf("Across %d random samples, the F-value (d1=%d, d2=%d) at the %fth percentile was %f", num_samples, d1, d2, percentile,quantile(sampled_Fs,percentile/100))

hist(sampled_Fs,breaks="FD",xlim=c(0, 10),xlab="F-value",col="skyblue2",main=paste(num_samples,"randomly sampled F-values under the\n ANOVA null hypothesis with d1 =", d1, "and d2 =",d2))

curve(df(x, d1, d2), from=0, to=10, xlab="F-value", ylab="Probability density",main=paste("PDF for F-distribution with d1 =", d1, "and d2 =",d2))