0
$\begingroup$

According to SelectKBest's documentation page, it 'select features according to the k highest scores', which in this case would be the Chi Square score.

https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html

If I understand this correctly, the selection does not involve the p-value - why?

According to my understanding, when we are doing a Chi Square test, we are doing a hypothesis testing with a null hypothesis that a given feature is independent from the target variable. If we can reject the null hypothesis, we can conclude that the two are dependent and thus having the given feature in the model would improve its performance.

We do this by calculating the Chi Square score for the feature. Given the degree of freedom, we can calculate the feature's p-value. If the score is below a predetermined alpha (usually 0.05), then the null hypothesis is rejected.

In other words, it seems that ultimately it is the p-value that matters in a Chi Square test. Sure, the k highest scores are likely to result in a p-value that is smaller than our predetermined alpha, but then isn't it possible to have a scenario where even the highest Chi Square score does not translate into a >alpha p-value?

PS: I read SelectKBest - Feature Selection - Python - SciKit Learn and it seems that SelectKBest does work as per my understanding, so it is all the more confusing

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

The documentation says

Select features according to the k highest scores.

which really explains the whole thing. The class is designed to select the $k$ highest scores, no more, no less.

The concept of a $p$-value is entirely orthogonal to whether or not the score is in the $k$ highest. Indeed, if we include the concept of $p$-values, then this gives rise to a number of additional questions, such as

  • What should the method do if none of the features have a $p$-value less than some threshold?
  • What should the method do if more than $k$ of the features have a $p$-value less than some threshold?

Selecting exactly $k$ features gives an obvious and consistent result. (The only exception is for ties: "Ties between features with equal scores will be broken in an unspecified way.")

If you want to use a $p$-value threshold as a feature selection method, you certainly can. You'll just need to use a different class.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am sorry but I must be misunderstanding some basic stuff. I thought the whole point of feature selection is to select features which are shown to be dependent with the target variable, so if we just choose features with respect to their ranking of Chi Square score, isn't it possible that we still can't conclude the highest ranked feature is dependent with the target variable, because its score corresponds to a p-value that does not allow us to reject the null-hypothesis (that the feature is dependent)? i.e. I don't understand how having high Chi square score alone guarantees dependence $\endgroup$
    – Constantly confused
    May 2, 2022 at 10:30
  • 1
    $\begingroup$ No feature selection method guarantees dependence. Even p-values have a false positive rate. $\endgroup$
    – Sycorax ♦
    May 2, 2022 at 10:32
1
$\begingroup$

Note that chi2 returns p values, but you don't even need the p value you just need the test statistic and degrees of freedom. From those two pieces of information alone we can determine if a result is statistically significant and can even compute if one sample has a smaller p value than another (assuming one of the two pieces of information are fixed between the two samples to compare)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.