fpr, fdr and fwe for feature selection

Scikit-learn's univariate feature selection module offers three similar sounding methods for feature selection

1. SelectFpr - false positive rate
2. SelectFdr - false discovery rate
3. SelectFwe - family-wise error rate

Only SelectFdr has a reference, which is to the Wikipedia. That article also covers fwe. I can't find a reference for Fpr, which I've seen used as a synonym for false discovery rate (for example here).

Can anyone link to or write a guide to choosing between these three methods? When is one desirable over another? Are they affected by the type of the data (continuous vs categorical, number of levels in a categorical variable), the distributions of either the predictors or the response, the correlations or dependence between features etc?

Those are among the methods for univariate feature selection.

1.13.2. Univariate feature selection

Univariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator. Scikit-learn exposes feature selection routines as objects that implement the transform method:

• [...]
• using common univariate statistical tests for each feature: false positive rate SelectFpr, false discovery rate SelectFdr, or family wise error SelectFwe.

Basically, univariate feature selection selects features in isolation. In the case of the three methods discussed, a p-value is computed from an ANOVA F-value or χ² statistic. From these value you can infer FPR or alternatively correct for FDR or FWER, then apply a threshold alpha, keeping only features whose corrected p-value are below said threshold.

• False positive rate: is the probability of falsely rejecting the null hypothesis (Wiki link)
• False discovery rate: is the expected rate of false rejections from all discoveries, i.e. all rejected hypotheses (Wiki link)
• Family wise error rate: is the probability of incurring at least one false positive among all discoveries (Wiki link)

Can anyone link to or write a guide to choosing between these three methods? When is one desirable over another? Are they affected by the type of the data (continuous vs categorical, number of levels in a categorical variable), the distributions of either the predictors or the response, the correlations or dependence between features etc?

Well, nothing is so simple. Basically, we can't know a priori which method works better for predictive power (which I suppose is your goal here).

From a practical point of view, FPR means no correction, so you keep the most features this way, FWER is the most conservative, so you are likely to keep less features with it, and FDR sits between both. It ends being a bit arbitrary, really, since the most common FWER corrections are likely to assume independence between hypotheses.

Are they affected by the type of the data (continuous vs categorical, number of levels in a categorical variable), the distributions of either the predictors or the response, the correlations or dependence between features etc?

The most common FDR-controling procedure (Benjamini-Hochberg) can accommodate positive dependence between hypotheses, but you're probably not really interested in the corrected p-values themselves, so I suggest you to try them all (in complete nested cross-validated sense) and pick the best performing one.

FPR is the probability of incurring a false positive, $P(FP)$. This is computed in isolation, and is ordinarily simply the p-value we get at a single hypothesis test. The thing is, when we compute multiple hypothesis, the probability of incurring false positives increase accordingly. So, we can control the rate these false discoveries occur, i.e. the FDR, which is simply $nFP/nD$, where $nFP$ is the expected number of false positives and $nD$ is the number of discoveries, i.e. the number of times we reject the null hypothesis. So, FDR-controlling procedures work on the whole space of hypotheses, not on any single one in isolation.