I have two related questions:
In the wikipedia definition of data dredging:
Data dredging (also data fishing, data snooping, and p-hacking) is the use of data mining to uncover patterns in data that can be presented as statistically significant, without first devising a specific hypothesis as to the underlying causality.
The process of data mining involves automatically testing huge numbers of hypotheses about a single data set by exhaustively searching for combinations of variables that might show a correlation.
I'm thinking of Random Forest when reading this definition. Although there is no hypothesis testing nor use of p-value, ... But does the predicted probability calculated by Random Forest make sense when it involve the using of data mining to find a patterns in data that can be significant in some extent?
IMHO: the predicted probability calculated by each single tree of RF is highly biased (because of data dredging), then RF tries to correct this bias by
- adding randomness to the model (variable selected at each nodes, bootstrapped training data sets)
- averaging the predicted probability of many trees.
But can this correct all the bias made by every single tree? If the correction of the predicted probability by this technique works? Can the same idea be applied to stepwise regression to correct its weakness cited here by @gung and @FrankHarrell?
This question is applied for other data mining methods as well.
If the predicted probability of Random Forest is considered "valid": when facing the imbalanced data, one way to improve the performance of RF is to use downsampling technique on the training data set before making trees (resampling the data in such a way that the positive and negative class are "balanced" in proportion). By doing this, the probability calculated is biased, because when the data is resampled, the proportion of positive/ negative observations (predicted probability) changes in each group due to resampling. To "correct" this biased probability, I imagine that one can not simply multiply the predicted probability by the prior downsampling ratio because even though the prior distribution is uniformly resampled, after the using of data mining to find the patterns in data (RF), the posterior distribution goes in different directions than the prior resampling ratio. Thus the multiplication of the posterior probability by the prior resampling ratio isn't valid anymore.
The question is: when using resampling technique, how to correct the predicted probability in Random Forest?
Don't hesitate to modify my question if it's not clear.