# Is there a risk of p-hacking when trying many models?

I am familiar with the concept of p-hacking for statistical tests, but does p-hacking apply to model building? Say you are trying many different linear models with a variety of variables, interactions, splines, polynomials, etc. What about trying different types of models, like starting with a basic linear regression model and then trying a lasso regression?

If p-hacking does apply, how to go about mitigating it?

• Trying more complex models can lead to the model overfitting on the data. P hacking is a kind of overfitting, so in a way yes this can happen. We combat the overfitting by using a validation procedure (cross validation or train/validation/test splits) Commented Apr 5, 2022 at 17:37
• Yes! If you are trying multiple models and then choosing the ones that seem to do what you want (like 'fit well' or worse 'produce a significant fit'), and follow that with inference (.e.g. testing a hypothesis) on the same data, then yes, that choice will impact the properties of your inferences, including the frequentist behavior of p-values (they will tend to be too low). This is indeed a form of p-hacking and it's astonishingly widely done by people that claim to be concerned about p-hacking. Commented Apr 6, 2022 at 1:55
• @Glen_b how does one know if they are entering this territory or not? Doesn't almost every modeler try out multiple iterations of their model to improve accuracy, etc. Commented May 9, 2022 at 21:30
• If you're looking at the same data you're using for estimation/testing/prediction to choose anything about your model, you're already there, unless your analysis has specifically been designed to incorporate the impact of such looking (this is almost never the case; you'll know if it is). There are ways of avoiding doing that looking at the same data. I suggest taking a look at Elements of Statistical Learning 2e as a fairly modern discussion of some of the issues, and perhaps Ch4 of Regression Modeling Strategies, which focuses on stepwise selection; the discussion is relevant widely Commented May 10, 2022 at 22:48
• Various explanations of aspects of this problem go back at least to the late 60s and probably longer. Almost every analysis in wide use that uses estimation testing or prediction assumes that the model is prespecified (up to unknown parameter values), and will be impacted by data-based model choice when it's done on the same data as used for the inference. Ideally, you use information external to the data for this, but if that's not possible, data splitting is generally a good choice. Commented May 10, 2022 at 22:52