10
$\begingroup$

I am learning about the feature selection using Python and SciKit learn. I came across the SelectKBest class, however it is unclear what kind of test is performed.

Select features according to the k highest scores.

http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest

The only reference for k-score I found was from the following wikipedia page.

Cohen's kappa coefficient is a statistic which measures inter-rater agreement for qualitative (categorical) items.

Is the k-score from SelectKBest class refers to the above? Thank you.

$\endgroup$

1 Answer 1

23
$\begingroup$

No, SelectKBest works differently.

It takes as a parameter a score function, which must be applicable to a pair ($X$, $y$). The score function must return an array of scores, one for each feature $X[:, i]$ of $X$ (additionally, it can also return p-values, but these are neither needed nor required). SelectKBest then simply retains the first $k$ features of $X$ with the highest scores.

So, for example, if you pass chi2 as a score function, SelectKBest will compute the chi2 statistic between each feature of $X$ and $y$ (assumed to be class labels). A small value will mean the feature is independent of $y$. A large value will mean the feature is non-randomly related to $y$, and so likely to provide important information. Only $k$ features will be retained.

Finally, SelectKBest has a default behaviour implemented, so you can write select = SelectKBest() and then call select.fit_transform(X, y) (in fact I saw people do this). In this case SelectKBest uses the f_classif score function. This interpretes the values of $y$ as class labels and computes, for each feature $X[:, i]$ of $X$, an $F$-statistic. The formula used is exactly the one given here: one way ANOVA F-test, with $K$ the number of distinct values of $y$. A large score suggests that the means of the $K$ groups are not all equal. This is not very informative, and is true only when some rather stringent conditions are met: for example, the values $X[:, i]$ must come from normally distributed populations, and the population variance of the $K$ groups must be the same. I don't see why this should hold in practice, and without this assumption the $F$-values are meaningless. So using SelectKBest() carelessly might throw out many features for the wrong reasons.

$\endgroup$
2
  • $\begingroup$ Got it. It is simply a scoring function that takes in a method/test as a parameter. Thanks! $\endgroup$
    – verkter
    Dec 30, 2016 at 16:45
  • $\begingroup$ hi just to confirm, for non normal data would you say chi2 is best? $\endgroup$
    – Maths12
    Feb 24, 2022 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.