Those are among the methods for univariate feature selection.
Univariate feature selection works by selecting the best features based on univariate statistical
tests. It can be seen as a preprocessing step to an estimator.
Scikit-learn exposes feature selection routines as objects that
implement the transform
method:
- [...]
- using common univariate statistical tests for each feature: false positive rate
SelectFpr
, false discovery rate SelectFdr
, or family
wise error SelectFwe
.
Basically, univariate feature selection selects features in isolation. In the case of the three methods discussed, a p-value is computed from an ANOVA F-value or χ² statistic. From these value you can infer FPR or alternatively correct for FDR or FWER, then apply a threshold alpha
, keeping only features whose corrected p-value are below said threshold.
- False positive rate: is the probability of falsely rejecting the null hypothesis (Wiki link)
- False discovery rate: is the expected rate of false rejections from all discoveries, i.e. all rejected hypotheses (Wiki link)
- Family wise error rate: is the probability of incurring at least one false positive among all discoveries (Wiki link)
Can anyone link to or write a guide to choosing between these three
methods? When is one desirable over another? Are they affected by the
type of the data (continuous vs categorical, number of levels in a
categorical variable), the distributions of either the predictors or
the response, the correlations or dependence between features etc?
Well, nothing is so simple. Basically, we can't know a priori which method works better for predictive power (which I suppose is your goal here).
From a practical point of view, FPR means no correction, so you keep the most features this way, FWER is the most conservative, so you are likely to keep less features with it, and FDR sits between both. It ends being a bit arbitrary, really, since the most common FWER corrections are likely to assume independence between hypotheses.
Are they affected by the type of the data (continuous vs categorical, number of levels in a categorical variable), the distributions of either the predictors or the response, the correlations or dependence between features etc?
The most common FDR-controling procedure (Benjamini-Hochberg) can accommodate positive dependence between hypotheses, but you're probably not really interested in the corrected p-values themselves, so I suggest you to try them all (in complete nested cross-validated sense) and pick the best performing one.
You asked:
can you elaborate a bit more on the difference between fpr and fdr. They are both rates related to the false rejection of the null hypothesis, but I don't understand how they differ.
FPR is the probability of incurring a false positive, $P(FP)$. This is computed in isolation, and is ordinarily simply the p-value we get at a single hypothesis test. The thing is, when we compute multiple hypothesis, the probability of incurring false positives increase accordingly. So, we can control the rate these false discoveries occur, i.e. the FDR, which is simply $nFP/nD$, where $nFP$ is the expected number of false positives and $nD$ is the number of discoveries, i.e. the number of times we reject the null hypothesis. So, FDR-controlling procedures work on the whole space of hypotheses, not on any single one in isolation.