2
$\begingroup$

I am familiar with the concept of p-hacking for statistical tests, but does p-hacking apply to model building? Say you are trying many different linear models with a variety of variables, interactions, splines, polynomials, etc. What about trying different types of models, like starting with a basic linear regression model and then trying a lasso regression?

If p-hacking does apply, how to go about mitigating it?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Trying more complex models can lead to the model overfitting on the data. P hacking is a kind of overfitting, so in a way yes this can happen. We combat the overfitting by using a validation procedure (cross validation or train/validation/test splits) $\endgroup$
    – Demetri Pananos
    Apr 5, 2022 at 17:37
  • 2
    $\begingroup$ Yes! If you are trying multiple models and then choosing the ones that seem to do what you want (like 'fit well' or worse 'produce a significant fit'), and follow that with inference (.e.g. testing a hypothesis) on the same data, then yes, that choice will impact the properties of your inferences, including the frequentist behavior of p-values (they will tend to be too low). This is indeed a form of p-hacking and it's astonishingly widely done by people that claim to be concerned about p-hacking. $\endgroup$
    – Glen_b
    Apr 6, 2022 at 1:55
  • $\begingroup$ @Glen_b how does one know if they are entering this territory or not? Doesn't almost every modeler try out multiple iterations of their model to improve accuracy, etc. $\endgroup$
    – haaris
    May 9, 2022 at 21:30
  • $\begingroup$ If you're looking at the same data you're using for estimation/testing/prediction to choose anything about your model, you're already there, unless your analysis has specifically been designed to incorporate the impact of such looking (this is almost never the case; you'll know if it is). There are ways of avoiding doing that looking at the same data. I suggest taking a look at Elements of Statistical Learning 2e as a fairly modern discussion of some of the issues, and perhaps Ch4 of Regression Modeling Strategies, which focuses on stepwise selection; the discussion is relevant widely $\endgroup$
    – Glen_b
    May 10, 2022 at 22:48
  • $\begingroup$ Various explanations of aspects of this problem go back at least to the late 60s and probably longer. Almost every analysis in wide use that uses estimation testing or prediction assumes that the model is prespecified (up to unknown parameter values), and will be impacted by data-based model choice when it's done on the same data as used for the inference. Ideally, you use information external to the data for this, but if that's not possible, data splitting is generally a good choice. $\endgroup$
    – Glen_b
    May 10, 2022 at 22:52

1 Answer 1

Reset to default
4
$\begingroup$

If you are trying out a variety of models for analysing the data then you are probably doing a preliminary study. There is no such thing as p-hacking in a preliminary study, but you will not be able to make any firm inferences until you have designed a secondary study of the interesting results and analysed new data. See this open access chapter for more details: https://link.springer.com/chapter/10.1007/164_2019_286#enumeration

On the other hand, if you are exploring a range of models in the hopes of finding a 'significant' result from one of them then you are greatly increasing the risk of false positive inferences exactly like "p-hacking".

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.