4
$\begingroup$

Is there a survey or some other resource that goes over commonly used methods for p-hacking in scientific literature?

I can broadly think of three ways -- 1) adjusting dataset, 2) adjusting reported metric, and 3) adjusting distribution of the null hypothesis, looking for more examples.

This is for a talk I'm giving on reproducibility crisis in ML research, looking for examples from other fields.

Improve this question
$\endgroup$
8
  • 4
    $\begingroup$ See stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf. $\endgroup$
    – user2974951
    Sep 26 at 14:59
  • 3
    $\begingroup$ Myriad ways researchers exercise choice during analysis. Various ways to apply "collect more data and try again" type methods. ... but I worry this will turn into a very open-ended "big-list" question. $\endgroup$
    – Glen_b
    Sep 26 at 15:42
  • 2
    $\begingroup$ I'd have thought in some areas not pre-specifying your question & looking at lots of different questions (either in terms of what the outcome is or what might influences it - e.g. nutrition data looking at all sorts of permutations of "Does chocolate/coffee/tea/wine/alocohol/vegetables/red meat/white meat/vegetarian diet/... increase/decrease risk of death/cancer/pregnancy outcomes/cardiovascular disease/Alzheimer's disease/...?") is a common problem. Adjusting the analysis model (e.g. linear regression with certain covariates vs. withoutout vs. non-parametric test) is of course also a thing. $\endgroup$
    – Björn
    Sep 26 at 15:56
  • 3
    $\begingroup$ ... model selection (forward or backward stepwise, or AIC-driven); other ways of including or excluding predictors; transformation of predictors (median split, binning, log, polynomial or spline transformations); including or excluding data points for this or that reason; deciding between different models (OLS vs. robust regression; Pearson vs. Spearman vs. Kendall correlation) based on the data rather than prior to collecting data; HARKing (hypothesizing after results are known) per @Björn. The possibilities are endless, and I would also be interested in such a reference. $\endgroup$
    – Stephan Kolassa
    Sep 26 at 16:17
  • 1
    $\begingroup$ @YaroslavBulatov Anything that changes the analysis will to a smaller or greater degree affect the p-value. Unless you deeply understand what you are doing, it will just randomly move it about, but that's the essence of p-hacking: try tons of stuff and take the random(ly) low (p-value). $\endgroup$
    – Björn
    Sep 29 at 7:36

1 Answer 1

Reset to default
7
$\begingroup$

A typical approach to answer such a question is to look it up at Wikipedia which has a broad article on this topic https://en.m.wikipedia.org/wiki/Data_dredging following the links in that article, and several others about problems with p-values shows that this is an incredibly broad topic with many statisticians having already said something about it.

The remaining problem seems to be about bringing order into all of it and classifying

commonly used methods for p-hacking

But given that the topic is broad and that the descriptions of methods can be vague, this might possibly not a great topic for a q&a here.

A problem with this question is that there are many ways of p-hacking, but nobody cares about generating a list of it.

An early work could be 'how to lie with statistics ' by Darrell Huff, but that book is far from sketching a complete overview of misuse of statistics.

The problem is that any statistical method can be abused. Transforming data, removing outliers, etc. they can be all used in wrong ways. And the question is not just asking for ways of p-hacking, but instead effectively asking for ways to perform statistical analysis in general.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the tip, I've bought the book. One example I recieved after giving my talk is the recent Alzeinheimer drug approval fiasco $\endgroup$
    – Yaroslav Bulatov
    Oct 2 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.