From my understanding, data mining bias occurs when someone repeatedly searches through a data set to find statistically significant results. How is this any different from p-hacking? What is the difference between p-hacking and data mining bias?
$\begingroup$
$\endgroup$
2
-
3$\begingroup$ Where is "data mining bias" defined? "Repeatedly search[ing] through a data set to find statistically significant results" doesn't make sense. Whether you are considering multiple combinations of variables, multiple subsets, or multiple models in a geometric series of statistical tests, the results are roughly the same: if you don't control for multiple testing, you inflate the type 1 error rate. $\endgroup$– AdamOMay 28, 2021 at 14:17
-
$\begingroup$ Related: stats.stackexchange.com/q/178873/176202 $\endgroup$– Frans RodenburgMay 28, 2021 at 15:29
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
This goes by many other names: Data dredging, fishing, selective inference, cherry picking, etc.
The main difference is that when you say $p$-hacking, you specifically talk about abuse of null-hypothesis significance testing. But $p$-values are not the only measure susceptible to this kind of misuse. You can find spurious associations of any kind in any moderately large data set.
I am not sure why there are so many different terms to describe the same problem, but likely, the latter here is meant to encompass a larger class of issues than just incorrect use of $p$-values.
answered May 28, 2021 at 14:21
$\begingroup$
$\endgroup$
Honestly, I never saw the "data mining bias" term before. Quick search gave me two references with definitions
Data-mining bias refers to an assumption of importance a trader assigns to an occurrence in the market that actually was a result of chance or unforeseen events. The data-mining bias, for many analysts, is considered an “insidious threat” because it can sneak up on traders and analysts alike during the research processes that lead traders and investors to make the plays they make in the market.
(https://corporatefinanceinstitute.com/resources/knowledge/other/data-mining-bias/)
Data mining bias occurs when investors go through a dataset in order to identify statistically significant patterns, which may come as a result of a random or unforeseen event. Therefore, data mining bias results in investment strategies that are unsuccessful in the long run. This type of bias usually occurs during the research process when investors try to put weight on identifying patterns.
(http://tech.harbourfronts.com/data-mining-bias/)
Seems like the term is used in a trading scenario, where finding spurious patterns in the data leads to bad investment decisions.
The term $p$-hacking is used in a science scenario, where a researcher switches his methodology, tools, or manipulates the data until achieving a "statistically significant" result that "proves" their hypothesis and is publishable. So "data mining bias" described above seems more related to accidentally finding spurious correlations in the data when doing exploratory analysis on big amounts of data. $P$-hacking is rather related to the actions of the analyst that are aimed at "making" the result statistically significant.
