3
$\begingroup$

Scikit-learn's univariate feature selection module offers three similar sounding methods for feature selection

  1. SelectFpr - false positive rate
  2. SelectFdr - false discovery rate
  3. SelectFwe - family-wise error rate

Only SelectFdr has a reference, which is to the Wikipedia. That article also covers fwe. I can't find a reference for Fpr, which I've seen used as a synonym for false discovery rate (for example here).

Can anyone link to or write a guide to choosing between these three methods? When is one desirable over another? Are they affected by the type of the data (continuous vs categorical, number of levels in a categorical variable), the distributions of either the predictors or the response, the correlations or dependence between features etc?

$\endgroup$

1 Answer 1

6
$\begingroup$

Those are among the methods for univariate feature selection.

1.13.2. Univariate feature selection

Univariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator. Scikit-learn exposes feature selection routines as objects that implement the transform method:

  • [...]
  • using common univariate statistical tests for each feature: false positive rate SelectFpr, false discovery rate SelectFdr, or family wise error SelectFwe.

Basically, univariate feature selection selects features in isolation. In the case of the three methods discussed, a p-value is computed from an ANOVA F-value or χ² statistic. From these value you can infer FPR or alternatively correct for FDR or FWER, then apply a threshold alpha, keeping only features whose corrected p-value are below said threshold.


  • False positive rate: is the probability of falsely rejecting the null hypothesis (Wiki link)
  • False discovery rate: is the expected rate of false rejections from all discoveries, i.e. all rejected hypotheses (Wiki link)
  • Family wise error rate: is the probability of incurring at least one false positive among all discoveries (Wiki link)

Can anyone link to or write a guide to choosing between these three methods? When is one desirable over another? Are they affected by the type of the data (continuous vs categorical, number of levels in a categorical variable), the distributions of either the predictors or the response, the correlations or dependence between features etc?

Well, nothing is so simple. Basically, we can't know a priori which method works better for predictive power (which I suppose is your goal here).

From a practical point of view, FPR means no correction, so you keep the most features this way, FWER is the most conservative, so you are likely to keep less features with it, and FDR sits between both. It ends being a bit arbitrary, really, since the most common FWER corrections are likely to assume independence between hypotheses.

Are they affected by the type of the data (continuous vs categorical, number of levels in a categorical variable), the distributions of either the predictors or the response, the correlations or dependence between features etc?

The most common FDR-controling procedure (Benjamini-Hochberg) can accommodate positive dependence between hypotheses, but you're probably not really interested in the corrected p-values themselves, so I suggest you to try them all (in complete nested cross-validated sense) and pick the best performing one.


You asked:

can you elaborate a bit more on the difference between fpr and fdr. They are both rates related to the false rejection of the null hypothesis, but I don't understand how they differ.

FPR is the probability of incurring a false positive, $P(FP)$. This is computed in isolation, and is ordinarily simply the p-value we get at a single hypothesis test. The thing is, when we compute multiple hypothesis, the probability of incurring false positives increase accordingly. So, we can control the rate these false discoveries occur, i.e. the FDR, which is simply $nFP/nD$, where $nFP$ is the expected number of false positives and $nD$ is the number of discoveries, i.e. the number of times we reject the null hypothesis. So, FDR-controlling procedures work on the whole space of hypotheses, not on any single one in isolation.

$\endgroup$
2
  • $\begingroup$ Thanks - can you elaborate a bit more on the difference between fpr and fdr. They are both rates related to the false rejection of the null hypothesis, but I don't understand how they differ. $\endgroup$
    – Dan
    Feb 13, 2018 at 14:21
  • $\begingroup$ if I allow FDR up to 0.5 this strikes me as intuitive and non-arbitrary because it seems to implement "I expect most of the selected features to be non-falsely-rejected" (in a non-predictive context). Is that right? Can we more parsimoniously say "non-spurious" instead of "non-falsely-rejected" or those different $\endgroup$ Oct 22, 2021 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.