4
$\begingroup$

If I have a large sample size, e.g. 100,000 data points, I know that most significance tests are going to come back with a very small p-value unless the null hypothesis is "true on the nose." In other words, even very small effects will be seen by the test. I can understand why this is true for a t-test, since when I compute the test statistic I have to divide by $\sqrt{n}$ in the formula for the standard error, so when $n$ is large my standard error is small, and so my t-statistic is huge. Is there a similar explanation for why an ANOVA F-test (let's say 1-way ANOVA) is likely to be significant when $n$ is large?

I'm asking so I can better explain things to my Stat 2 class. When asked in class today, the explanation I tried was that, when $n$ is huge $MSE$ is going to be very small (because it's $SSE/(n-k)$), so the $F$-statistic will be huge. The students followed up by asking why the large df in the $F$-statistic doesn't account for this and so give reasonable $p$-values even for very large $F$-statistics (rather than ultra small $p$-values as we've been seeing in our examples).

I know, of course, that for a two-sample t-test $F = t^2$, so I can deduce significance as a special case of the reasoning above, but I'm more interested in the general case of more than 2 groups, and an explanation that doesn't require the derivation that $F = t^2$. Any help would be much appreciated. Thanks!

$\endgroup$
4
  • 2
    $\begingroup$ This question, generalized and posed slightly differently, appears at stats.stackexchange.com/questions/2516. The common spirit is to ask why having more data gives one more power to reject a false null hypothesis. So, rather than focusing on the F-test itself, you might consider discussing this general issue: your students might learn much more for the same effort. $\endgroup$
    – whuber
    Mar 31, 2016 at 18:16
  • 1
    $\begingroup$ @whuber. I read that thread just before posting, but could not distill from it an explanation that would satisfy my students. Still, it inspired me to prepare a class on effect size that I'm delivering tomorrow. By the way, I'm a big fan of your answers here. Thanks for posting! $\endgroup$ Mar 31, 2016 at 22:59
  • $\begingroup$ the explanation I tried was that, when n is huge MSE is going to be very small Sure, you're dividing by a huge $n$, but you're also getting a huge SSE. $\endgroup$
    – Dave
    Jan 16 at 20:05
  • $\begingroup$ Yep, that was literally the next sentence in the question "The students followed up by asking..." Also, the third (last) comment I left below Michael's answer. $\endgroup$ Jan 16 at 20:24

1 Answer 1

1
$\begingroup$

To use the usual arguments why small effects mean low p-values if the sample size is large (like in the link provided by @whuber), you need some measure of effect size in ANOVA. A simple one (which also works for the general normal linear model) is the sample R-squared $R^2$. It measures the proportion of explained variability in the response accounted for by the covariables (in one-way ANOVA the grouping factor) and estimates some "true" proportion $\theta$.

So we can say: For large samples, even if $R^2$ is close to zero, the F-test will provide a small p-value.

Illustration by simulation:

# Input
set.seed(20)
n <- 1000000
x <- sample(LETTERS[1:3], n, replace = TRUE)
y <- 2 + 0.01 * (x == "B") - 0.01 * (x == "C") + rnorm(n)
fit <- lm(y ~ x)

summary(fit)

# Output (partial)
Residual standard error: 1.001 on 999997 degrees of freedom
Multiple R-squared:  9.653e-05, Adjusted R-squared:  9.453e-05 
F-statistic: 48.27 on 2 and 999997 DF,  p-value: < 2.2e-16

Illustration by math: The F-statistic is a simple function of $R^2$ and one can show that under the assumptions of the normal linear model and under the null hypothesis of the F-test, $R^2$ has a beta distribution (see What is the distribution of $R^2$ in linear regression under the null hypothesis? Why is its mode not at zero when $k>3$?) with $$ E(R^2) = \frac{k-1}{n-1} $$ and $$ Var(R^2) = \frac{1}{4(n-k+1)} $$ ($k$ is the number of parameters of the model, e.g. the number of groups in a 1-way-ANOVA). So working with $R^2$ and the beta distribution is equivalent to working with F-statistic and F-distribution. For large $n$ and fixed $k$, the beta distribution above concentrates about 0, leading to a small p-value for any non-zero observed $R^2$.

$\endgroup$
5
  • $\begingroup$ I also read about $\eta^2$ as a measurement of effect size for ANOVA. Is $R^2$ preferred over $\eta^2$? How about $\omega^2$? Thanks! $\endgroup$ Mar 31, 2016 at 23:00
  • $\begingroup$ Eta-squared is the same as R-squared, so it is up to you. No idea if a test on omega-squared is equivalent to a test on R-squared... $\endgroup$
    – Michael M
    Apr 1, 2016 at 6:20
  • $\begingroup$ Okay, so I agree that as $n \to \infty$, I expect $R^2 \to 0$. So SSM/SST is going to zero, hence SSE/SST must be going to 1. Wouldn't this cause an F-statistic that goes to zero? But in practice with these large datasets, we've been seeing large F statistics, not small ones. The class and I thought it had to do with the large degrees of freedom of the denominator. Are our findings at odds with your answer? $\endgroup$ Apr 1, 2016 at 11:00
  • $\begingroup$ Under the null hypothesis, the expectation of the corresponding F-statistic approaches 1 for large $n$, so something in your reasoning might be wrong. A large F-statistic means strong evidence against the null. But is it really a strong effect? This is the tricky part and the reason why I was going via R-squared. $\endgroup$
    – Michael M
    Apr 1, 2016 at 11:58
  • $\begingroup$ Thanks. One more question. In your simulation, the value of the test statistic is 48.27. Each time I make n larger by a degree of magnitude, F gets larger by a degree of magnitude. Is there a theorem to justify that? I already don't see why it's true for a t-distribution. The test statistic in that case is $t = (x-\overline{x})/(s/\sqrt{n})$, but as $n$ increases both $s$ and $\sqrt{n}$ increase. Why should the denominator be going to zero (which seems necessary for making the test statistic go to $\infty$)? Thanks! $\endgroup$ Apr 1, 2016 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.