Why is ANOVA not p-hacking?

Question

Say we have some data with many parameters. As an example let's say I'm an not-so-ethical journalist working for a food website and I'm looking to write some clickbait article "backed by science" about how some food or lifestyle is good/bad for you.

My data might contain a set of thousands of people, their socioeconomic status, what they had for breakfast, whether they're vegetarian, whether they prefer tea or coffee, etc. and also what their scores are for an IQ test.

If I split the dataset into many (100+) groups, where group one might split vegetarians/meat eaters, group two might split coffee drinkers and tea drinkers, group 3 might split female coffee drinkers and male tea drinkers etc. If then use ANOVA to compare all groups to see if there's a statistically significant difference, is this not effectively p-hacking?

I understand that ANOVA won't tell us which groups are different, but it seems like it would be a quick way to confirm that there IS a (false) positive in there. And then it's a case of finding out which groups are different.

Answer 1

It is not p-hacking. Multiple groups are compared but only a single hypothesis is tested.

ANOVA computes a ratio of variances and a p-value can be computed for that ratio based on a single hypothesis.

Possibly the idea of 'anova = p-hacking' arises due to the confusing aspect that a hypothesis test is often not used to test a null hypothesis, but instead to give prove/justification for the alternative hypothesis (which can be many at once).

Note that ANOVA has the following properties

• ANOVA doesn't tell which of the many groups are different, but only that they are not the same.
• ANOVA is less powerfull than seperate t-test for all sort of combinations of groups.

If I split the dataset into many (100+) groups, where group one might split vegetarians/meat eaters, group two might split coffee drinkers and tea drinkers, group 3 might split female coffee drinkers and male tea drinkers etc. If then use ANOVA to compare all groups to see if there's a statistically significant difference, is this not effectively p-hacking?

It is not clear how you are gonna do this splitting and ANOVA. With ANOVA you can have multiple groups, but these should not be overlapping. You might do something like a linear model with all those variables, but then each additional group/variable is gonna decrease the degrees of freedom and make the ANOVA test less sensitive/powerful.

Answer 2

You appear to perhaps be confusing p hacking with application of statistical methods involving more than a single binary explanatory variable (also see below, where I address post hoc tests following ANOVA). This reminds me of the accusation "Anyone can lie with statistics!" which, while true, does not reveal some special truth about statistics: when describing human communication you can just expostulate "Anyone can lie" full stop.

Regardless of statistical method (e.g., ANOVA, or something else), the term p hacking connotes a researcher (or research group) who has prioritized reporting "statistically significant findings" over reporting "scientifically substantive findings" (whether positive or negative)… if not outright abandoning the reporting of findings of scientific substance. So p hacking is less about a specific statistical method (ANOVA or otherwise), than it is an orientation towards what one does with the method (and the study design, measurement, selection of participants, level of inference, etc.).

A good cue, if you are unsure whether you are p hacking, is whether your research is lead by a well-formed theory-driven question in your particular domain of research, one that it would be valuable to answer, (with choice of research methods, including statistical methods following the lead of that question), or whether you start with a specific statistical method and go looking for "questions" to answer.

Finally, ANOVA (and other omnibus hypothesis tests) do just what they say on the package: identify whether there is enough deviation from similarity among all the groups collectively for a given type I error rate (i.e. $\alpha$). Of course, p hacking really comes into play with the post hoc pairwise tests—if there are $k$ groups, then there are as many as $\frac{k^2 - k}{2}$ of these, which gives a good chance (specifically an $\alpha$ chance) of false positives to be reported as "statistically significant". Non-p hacking researchers can account for this by using muliple comparisons adjustment strategies (e.g., false discovery rate, family-wise error rate, etc.).

Randall Munroe's "Significant" is always worth a chuckle, and is germane, so I am including yet again here. :)

Answer 3

There is no doubt that ANOVA--and any many other statistical tests--can be used for P-hacking, if used repeatedly on many random datasets, as you describe.

At the 5% level, about 1 in 20 of the tests will have a P-value below 0.05.

Under $H_0$ for an exact test based on a continuous test statistic, the distribution of the P-value is standard uniform, and thus has probability 0.05, below 0.05.

Shown below are simulations for (a) t.tests with normal data, (b) exact binomial tests with binomial counts, and (c) one-sample Wilcoxon tests with normal data.

When $H_0$ is true: Only the t.test shows a standard uniform distribution for the P-value. The binomial test has P-values as close to 5% as possible (without exceeding 5%), so there is no (nonrandomized) binomial test at exactly the 5% level. The Wilcoxon test is based on ranks; hence the test statistic is not continuous. But a test near the 5% level is available.

R code for figure:

set.seed(1234)
pv.t = replicate(10^5, 
         t.test(rnorm(25, 100, 15), mu = 100)$p.val)
mean(pv.t <= 0.05)
[1] 0.05017       # 5% within margin of sim error


set.seed(1235)
pv.b = replicate(10^5, 
        binom.test(rbinom(1, 10, .5), 10)$p.val)
mean(pv.b <= 0.05)
[1] 0.02173     # No test at exactly 5% available
                #  This test rejects for x=0,1,9, or 10
set.seed(1236)
pv.w = replicate(10^5, 
        wilcox.test(rnorm(25, 100, 15), mu = 100)$p.val)           
mean(pv.w <= 0.05)
[1] 0.04905       # test very nearly at 5%

par(mfrow=c(1,3))
 hist(pv.t, prob=T, col="skyblue2", main="T Test")
  curve(dunif(x), add=T, col="brown", lwd=2)
  abline(v=.05, col="red", lty="dotted", lwd=2)
 hist(pv.b, prob=T, col="skyblue2", main="Exact Binomial Test")
  curve(dunif(x), add=T, col="brown", lwd=2)
  abline(v=.05, col="red", lty="dotted", lwd=2)
 hist(pv.w, prob=T, col="skyblue2", main="Wilcoxon Test")
  curve(dunif(x), add=T, col="brown", lwd=2)
  abline(v=.05, col="red", lty="dotted", lwd=2)
par(mfrow=c(1,1))