Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
"P-hacking", "fishing", and "garden of forking paths" as explained here and here describes an exploratory data analysis-like style of doing research that produces biased estimates.
Does testing model assumptions (e.g. normality, homoskedasticity in regression) using statistical tests on the same data set that is used to fit the model considered a "p-hacking" or "garden of forking paths" problem?
The results of those tests certainly affect what model the researcher ultimately choices to fit.
It's not quite the same thing in the sense that the practice of testing whether assumptions were violated was originally intended to make sure an appropriate analysis was done, but as it turns out, it does have some of the same consequences (see e.g. this question). But it is in a milder form than the more extreme variants of p-hacking that are specifically targeted at somehow getting the p-value for the effect of interest below 0.05. That is unless you start combining multiple problematic practices (e.g. checking for normality, checking for homoscedasticity, checking covariates that "should" be in the model, checking for linearity of covariates, checking interactions etc.). I am not sure whether anyone has looked into how much that invalidates the final analysis.
Of course the other issue is that testing for normality is not normally meaningful (see e.g. this discussion). For small sample sizes you do not reliably pick up massive deviations that truly violate your assumptions, while for large sample sizes e.g. the t-test becomes quite robust to deviations but the normality test will start to detect tiny deviations that do not matter. It is much better to (whenever possible) specify an appropriate model based on previous data or subject matter knowledge. When that is not possible, it may be best to use methods that are more robust to violations of distributional assumptions or have none/fewer.
I do not believe that checking the assumptions of any model qualifies as p-hacking /fishing. In the first article, the author is talking about analysts who are repeatedly performing analyses on a data set and only reporting the best result. In other words, they are purposely portraying a biased picture of what is happening in the data.
Testing the assumptions of regression or any model is mandatory. What is not mandatory is to repeatedly re-sample from the data in order to ascertain the best possible outcome. Assuming researchers have a large enough sample to pull from, they will sometimes re-sample over and over again...perform hypothesis tests over and over again....until they achieve the result they want. Hence p-hacking. They're hacking the p-value via looking for the desired result and won't quit until they find it (fishing). So even if out of 100 hypothesis tests they only achieve 1 with a significant result, they'll report the p-value belonging to that particular test and omit all the others.
Does this make sense? When checking model assumptions, you're making sure that the model is appropriate for the data that you have. With p-hacking/fishing, you are endlessly searching the data/manipulating the study in order to achieve your desired outcome.
As for the purpose of multiple comparison, if you keep running a model through the mud endlessly trying to find a way to invalidate it (or validate it) then eventually you will find a way. This is fishing. If you want to validate a model, then you'll find a way. If you want to invalidate it, then you'll find a way. The key is to have an open mind and find out the truth - not just see what you want to see.
