8
$\begingroup$

"P-hacking", "fishing", and "garden of forking paths" as explained here and here describes an exploratory data analysis-like style of doing research that produces biased estimates.

Does testing model assumptions (e.g. normality, homoskedasticity in regression) using statistical tests on the same data set that is used to fit the model considered a "p-hacking" or "garden of forking paths" problem?

The results of those tests certainly affect what model the researcher ultimately choices to fit.

$\endgroup$
5
  • 1
    $\begingroup$ See Does testing for assumptions affect type I error?. $\endgroup$ Jul 21, 2017 at 21:46
  • 5
    $\begingroup$ testing assumptions doesn't do a thing. But people rarely just test, the outcome of the test influences what they do next... and it's the "what they do next" that's the issue. It's when the outcome of the test changes what you would do that there's a "fork". It's important to make sure you're asking the right thing there. $\endgroup$
    – Glen_b
    Jul 22, 2017 at 2:05
  • 3
    $\begingroup$ @Glen_b excellent point - for example, if you test data for normality, and then choose what test to use depending on whether the null was accepted or not, you're essentially using a composite test of unknown properties (Type I & II errors). $\endgroup$
    – DeltaIV
    Jul 22, 2017 at 12:28
  • $\begingroup$ @Glen_b, that is exactly what I'm wondering about. I'm not sure if choosing a model based on the results of tests of assumptions would bias an estimator (I'm trying to think of an example of that), but it would definitely affect the standard errors. $\endgroup$ Jul 25, 2017 at 14:15
  • $\begingroup$ @scortchi, that is a great example of what I'm concerned about. Yet, testing assumptions or at least evaluating them based on the data used to fit the model seems to be standard practice. That's how I have been taught in all my classes. $\endgroup$ Jul 25, 2017 at 14:19

2 Answers 2

3
$\begingroup$

It's not quite the same thing in the sense that the practice of testing whether assumptions were violated was originally intended to make sure an appropriate analysis was done, but as it turns out, it does have some of the same consequences (see e.g. this question). But it is in a milder form than the more extreme variants of p-hacking that are specifically targeted at somehow getting the p-value for the effect of interest below 0.05. That is unless you start combining multiple problematic practices (e.g. checking for normality, checking for homoscedasticity, checking covariates that "should" be in the model, checking for linearity of covariates, checking interactions etc.). I am not sure whether anyone has looked into how much that invalidates the final analysis.

Of course the other issue is that testing for normality is not normally meaningful (see e.g. this discussion). For small sample sizes you do not reliably pick up massive deviations that truly violate your assumptions, while for large sample sizes e.g. the t-test becomes quite robust to deviations but the normality test will start to detect tiny deviations that do not matter. It is much better to (whenever possible) specify an appropriate model based on previous data or subject matter knowledge. When that is not possible, it may be best to use methods that are more robust to violations of distributional assumptions or have none/fewer.

$\endgroup$
4
$\begingroup$

I do not believe that checking the assumptions of any model qualifies as p-hacking /fishing. In the first article, the author is talking about analysts who are repeatedly performing analyses on a data set and only reporting the best result. In other words, they are purposely portraying a biased picture of what is happening in the data.

Testing the assumptions of regression or any model is mandatory. What is not mandatory is to repeatedly re-sample from the data in order to ascertain the best possible outcome. Assuming researchers have a large enough sample to pull from, they will sometimes re-sample over and over again...perform hypothesis tests over and over again....until they achieve the result they want. Hence p-hacking. They're hacking the p-value via looking for the desired result and won't quit until they find it (fishing). So even if out of 100 hypothesis tests they only achieve 1 with a significant result, they'll report the p-value belonging to that particular test and omit all the others.

Does this make sense? When checking model assumptions, you're making sure that the model is appropriate for the data that you have. With p-hacking/fishing, you are endlessly searching the data/manipulating the study in order to achieve your desired outcome.

As for the purpose of multiple comparison, if you keep running a model through the mud endlessly trying to find a way to invalidate it (or validate it) then eventually you will find a way. This is fishing. If you want to validate a model, then you'll find a way. If you want to invalidate it, then you'll find a way. The key is to have an open mind and find out the truth - not just see what you want to see.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.