According to SelectKBest's documentation page, it 'select features according to the k highest scores', which in this case would be the Chi Square score.
https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html
If I understand this correctly, the selection does not involve the p-value - why?
According to my understanding, when we are doing a Chi Square test, we are doing a hypothesis testing with a null hypothesis that a given feature is independent from the target variable. If we can reject the null hypothesis, we can conclude that the two are dependent and thus having the given feature in the model would improve its performance.
We do this by calculating the Chi Square score for the feature. Given the degree of freedom, we can calculate the feature's p-value. If the score is below a predetermined alpha (usually 0.05), then the null hypothesis is rejected.
In other words, it seems that ultimately it is the p-value that matters in a Chi Square test. Sure, the k highest scores are likely to result in a p-value that is smaller than our predetermined alpha, but then isn't it possible to have a scenario where even the highest Chi Square score does not translate into a >alpha p-value?
PS: I read SelectKBest - Feature Selection - Python - SciKit Learn and it seems that SelectKBest does work as per my understanding, so it is all the more confusing