Skip to main content

Questions tagged [p-hacking]

Misuse of data analysis to find patterns that can be presented as statistically significant. Done by performing many statistical tests and only reporting those that have yielded significant results.

Filter by
Sorted by
Tagged with
1 vote
1 answer
57 views

Does division into training, validation, and test sets make p-hacking impossible?

Is it possible to commit "statistical sins" such as stopping data exploration if an analysis yields a significant p-value deciding whether to include or drop outliers postanalyses when ...
Sam's user avatar
  • 621
20 votes
5 answers
2k views

Having a second bite of the data-apple without p-hacking

My hypothesis concerns intervention versus control in a randomised controlled trial (between-subjects, n=500 per group, online survey experiment). I pre-registered that my primary test would be a ...
Amorphia's user avatar
  • 913
2 votes
1 answer
38 views

Is it possible to test if the response difference over two time points is significantly different between groups when there are no repeated subjects?

I currently have a dataset where I've collected data between two groups (disease versus control). The experimental design is to evaluate how the response variable changes over time. Thus, we have 4 ...
Syuma's user avatar
  • 125
0 votes
0 answers
5 views

Designing a random factor in a linear mxied effect model to combine repeated and single exposure datasets to prevent p-hacking

I currently have data that can be split into two categories: one is a repeated measures data (collected at multiple time points), whilst another is a single measure dataset (one single time point out ...
Syuma's user avatar
  • 125
2 votes
1 answer
42 views

Is It valid to use a Linear Mixed effect Model to quantify a group 'summary' value and plot it when the factors are all categorical?

I currently have a dataset with two factors: Gene and Timepoint. Both of these factors are categorical in nature where Gene has 2 levels defined as control vs disease, and timepoint has 4 levels: ...
Syuma's user avatar
  • 125
2 votes
1 answer
65 views

P-Hacking concern: How valid is it to take statistically significant linear models from two different datasets and compare their coefficients?

I was wondering what if the following process can be considered p value hacking or just bad statistical practice. What I currently have are two subsets of one large parent dataset. These subsets are ...
Syuma's user avatar
  • 125
2 votes
0 answers
46 views

Does applying a variable transformation to experiment data result in p-hacking?

From formal statistics classwork and some research experience, I understand that it is widely considered standard practice to apply transformations to predictor variables for conducting significance ...
rdos84's user avatar
  • 21
4 votes
1 answer
77 views

Split-sample analysis as a way of avoiding p-hacking?

In my lab we pre-register our hypotheses prior to running an experiment, and we also attempt to fully specify our exclusion criteria, coding practices for small categories and edge cases, etc. However,...
octern's user avatar
  • 624
4 votes
1 answer
223 views

Fine-tuning of methods before main analysis: p-value hacking?

I am reviewing a paper for a journal. The authors propose a new classification scheme. They then try to assess the classification accuracy using some novel predictor data. Standard classification ...
user7831861's user avatar
2 votes
1 answer
109 views

Does p-hacking affect the Type II (False Negative) error rate?

I know that p-hacking (also know as data dredging) increases the Type I error rate as exemplified by this XKCD example. My question is whether it influences the Type II error rate. I'm not sure ...
sloth's user avatar
  • 123
2 votes
1 answer
126 views

Double dipping/p-hacking and interim sample size reestimation

When one applies interim sample size reestimation based on nuisance parameter estimates (observe fraction of initial sample size, estimate nuisance parameter(s) based on the obtained data, reestimate ...
gir emee's user avatar
2 votes
0 answers
35 views

Dishonest approach to multiple testing

Has the following --dishonest-- approach been discussed somewhere? Has it a name? I am trying to grasp what intrinsic assumption it violates but I am struggling to find the relevant keywords for the ...
ocram's user avatar
  • 22.1k
4 votes
1 answer
520 views

Is there a risk of p-hacking when trying many models?

I am familiar with the concept of p-hacking for statistical tests, but does p-hacking apply to model building? Say you are trying many different linear models with a variety of variables, interactions,...
haaris's user avatar
  • 61
10 votes
3 answers
3k views

Why is ANOVA not p-hacking?

Say we have some data with many parameters. As an example let's say I'm an not-so-ethical journalist working for a food website and I'm looking to write some clickbait article "backed by science&...
gazm2k5's user avatar
  • 315
11 votes
3 answers
4k views

Is this p-hacking?

I'm currently looking into the gender pay gap using data from glass door (found via kaggle). The dataset has columns for gender, age, performance evaluations of employees, seniority, pay etc. For ...
gazm2k5's user avatar
  • 315
3 votes
1 answer
63 views

Example of multiple comparisons problem

Pharma companies can get FDA approvals on multiple indications for a single drug, e.g. different kinds of cancers. I was reading about the multiple comparisons problem, i.e. an effective inflation of ...
Golden_Ratio's user avatar
11 votes
2 answers
668 views

Increasing sample size to obtain width of CI / SE: p-hacking?

I'm involved in running experiments, where we want to obtain sufficient sample size to obtain a certain width of CI (or equivalently a certain power). We currently run a pilot, of a few hundred units, ...
Jeremy Miles's user avatar
  • 18.2k
3 votes
2 answers
520 views

What is the difference between p-hacking and data mining bias?

From my understanding, data mining bias occurs when someone repeatedly searches through a data set to find statistically significant results. How is this any different from p-hacking? What is the ...
Flux's user avatar
  • 145
5 votes
1 answer
116 views

Stop collecting data when our confidence interval is sufficiently narrow?

I am running an experiment that is unpleasant and time-consuming, so I want to minimize the number of times I have to do it. I have done a sample size calculation to determine that I need $100$ ...
Dave's user avatar
  • 64.2k
1 vote
0 answers
156 views

Data dredging in machine learning by training multiple models

Many machine learning papers I read follow something like this procedure: Split into test and train set, train different models on the train set and evaluate their performance on the test set, report ...
PascalIv's user avatar
  • 829
4 votes
1 answer
1k views

Bonferroni correction versus the F-test

Many sources emphasize the importance of the F-test p-value in multiple regression and justify this in terms of p-hacking. It's kind of intuitive that if you can't reject the null hypothesis that all ...
zkurtz's user avatar
  • 2,160
0 votes
1 answer
205 views

Is adding batches of samples to a non-stat sig sample p-hacking?

Here's my scenario. I take 1000 samples each from my control and treatment. I note that the difference in means is not statistically significant (stat-sig) under the null hypothesis testing. I then ...
Data Max's user avatar
3 votes
1 answer
129 views

Does running multiple similar models lead to p-hacking?

In my analysis I am testing a lot of similar models. I have two variables (let´s call them A and B) and a whole bunch of other variables (C1....C10). All models have the same approach: It is always ...
heyho's user avatar
  • 103
5 votes
1 answer
389 views

ROC-style curves for calculating sample size, power, alpha, and effect size

I found an awesome R package called pwr that does all sorts of calculations about sample sizes, power, effect sizes, and so on, and I've been playing. I have a ...
Dave's user avatar
  • 64.2k
2 votes
1 answer
420 views

Does principal component analysis (PCA) lead to p-hacking?

My knowledge of principal component analysis is only conceptual; I know nothing about the nuts and bolts of how it works. I learned about it from its use in sociolinguistics, as in Horvath & D. ...
joshisanonymous's user avatar