Questions tagged [p-hacking]
Misuse of data analysis to find patterns that can be presented as statistically significant. Done by performing many statistical tests and only reporting those that have yielded significant results.
18
questions
4
votes
1
answer
203
views
Fine-tuning of methods before main analysis: p-value hacking?
I am reviewing a paper for a journal. The authors propose a new classification scheme. They then try to assess the classification accuracy using some novel predictor data. Standard classification ...
- 105
0
votes
0
answers
25
views
Local regression to overcome autocorrelation?
I'm facing a very similar situation to the question found here. I'd like to test whether the slope of time series data changes at a particular point and I'm aiming for a simple test.
The proposed ...
- 154
2
votes
1
answer
85
views
Does p-hacking affect the Type II (False Negative) error rate?
I know that p-hacking (also know as data dredging) increases the Type I error rate as exemplified by this XKCD example.
My question is whether it influences the Type II error rate. I'm not sure ...
- 123
2
votes
1
answer
108
views
Double dipping/p-hacking and interim sample size reestimation
When one applies interim sample size reestimation based on nuisance parameter estimates (observe fraction of initial sample size, estimate nuisance parameter(s) based on the obtained data, reestimate ...
- 33
2
votes
0
answers
30
views
Dishonest approach to multiple testing
Has the following --dishonest-- approach been discussed somewhere? Has it a name?
I am trying to grasp what intrinsic assumption it violates but I am struggling to find the relevant keywords for the ...
- 21k
2
votes
1
answer
267
views
Is there a risk of p-hacking when trying many models?
I am familiar with the concept of p-hacking for statistical tests, but does p-hacking apply to model building? Say you are trying many different linear models with a variety of variables, interactions,...
- 41
10
votes
3
answers
2k
views
Why is ANOVA not p-hacking?
Say we have some data with many parameters. As an example let's say I'm an not-so-ethical journalist working for a food website and I'm looking to write some clickbait article "backed by science&...
- 305
10
votes
3
answers
3k
views
Is this p-hacking?
I'm currently looking into the gender pay gap using data from glass door (found via kaggle). The dataset has columns for gender, age, performance evaluations of employees, seniority, pay etc.
For ...
- 305
3
votes
1
answer
55
views
Example of multiple comparisons problem
Pharma companies can get FDA approvals on multiple indications for a single drug, e.g. different kinds of cancers.
I was reading about the multiple comparisons problem, i.e. an effective inflation of ...
- 503
11
votes
2
answers
573
views
Increasing sample size to obtain width of CI / SE: p-hacking?
I'm involved in running experiments, where we want to obtain sufficient sample size to obtain a certain width of CI (or equivalently a certain power).
We currently run a pilot, of a few hundred units, ...
- 15.6k
3
votes
2
answers
356
views
What is the difference between p-hacking and data mining bias?
From my understanding, data mining bias occurs when someone repeatedly searches through a data set to find statistically significant results. How is this any different from p-hacking? What is the ...
- 145
5
votes
1
answer
87
views
Stop collecting data when our confidence interval is sufficiently narrow?
I am running an experiment that is unpleasant and time-consuming, so I want to minimize the number of times I have to do it. I have done a sample size calculation to determine that I need $100$ ...
- 50.6k
1
vote
0
answers
126
views
Data dredging in machine learning by training multiple models
Many machine learning papers I read follow something like this procedure:
Split into test and train set,
train different models on the train set and evaluate their performance on the test set,
report ...
- 745
4
votes
1
answer
814
views
Bonferroni correction versus the F-test
Many sources emphasize the importance of the F-test p-value in multiple regression and justify this in terms of p-hacking. It's kind of intuitive that if you can't reject the null hypothesis that all ...
- 2,140
0
votes
1
answer
125
views
Is adding batches of samples to a non-stat sig sample p-hacking?
Here's my scenario. I take 1000 samples each from my control and treatment. I note that the difference in means is not statistically significant (stat-sig) under the null hypothesis testing. I then ...
- 31
3
votes
1
answer
93
views
Does running multiple similar models lead to p-hacking?
In my analysis I am testing a lot of similar models. I have two variables (let´s call them A and B) and a whole bunch of other variables (C1....C10).
All models have the same approach: It is always ...
- 93
5
votes
1
answer
268
views
ROC-style curves for calculating sample size, power, alpha, and effect size
I found an awesome R package called pwr that does all sorts of calculations about sample sizes, power, effect sizes, and so on, and I've been playing.
I have a ...
- 50.6k
2
votes
1
answer
299
views
Does principal component analysis (PCA) lead to p-hacking?
My knowledge of principal component analysis is only conceptual; I know nothing about the nuts and bolts of how it works. I learned about it from its use in sociolinguistics, as in Horvath & D. ...
- 385