0
$\begingroup$

When looking for correlation between features (for feature selection), I found that sklearn implementation of Chi2 test of independence produce significantly different results from scipy.stats implementation.

My data contains 300 records, with 6 anonymized categorical features and the label. My focus is on the feature A. This data is available here in github. In the folder, see the file sample300.csv, alongside the notebook with my testing code.

For the feature A, sklearn's SelectKBest() returned the lowest ranking, suggesting there is no correlation between A and label. But scipy.stats.chi2_contingency() returned very different result, suggesting the correlation is very high.

Because of mismatch between the two, I went a long way performing a number of different tests described in detail in this article The results suggest that the scipy implementation is correct, while sklearn implementation is incorrect.

This conclusion surprising, given the popularity of sklearn.feature_selection_SelectKBest. Why are the two implementations of chi2 producing contradicting results? Did I make a methodical error somewhere or is this really a bug? I am looking forward for some critical peer review.

$\endgroup$
0
$\begingroup$

You should go back to the literature on Chi2 tests and about what they can tell you, and in this case what they cannot tell you. It's a reasoning issue in your case.

Indeed, the test will tell you that there is a strong correlation between experience and success. But that it doesn't tell you there isn't one between education and succes doesn't necessarily mean there isn't one. You are working under the assumption that education and experience are the only determinant factors of success, which is wrong. You are not controlling for all other factors that are predictors of success, some of which might not be observable (e.g., personality traits such as conscienciousness). Many of them are strongly correlated to education and as long as you are not taking them into account, you can say nothing about the importance of education in success.

In order words, you are using this python-package and the test it uses in the wrong way.

Update after comment : But scipy and sklearn implement chi2 in slightly different ways to begin with. This is a known difference between the two methods. However, this does not give huge differences between the results. (https://stackoverflow.com/questions/50932433/scipy-and-sklearn-chi2-implementations-give-different-results).

Whether A and B are education and experience or not, you should still be careful to control for all variables.

$\endgroup$
2
  • $\begingroup$ I appreciate this answer. This tells me clearly that I might have misled some readers so I want to add this clarification: The original meaning of feature A does not matter. In the article (meant for general public) I called this feature 'education' to build a story. But here let's just call it feature A and forget about its business meaning. The meaning does not matter. The question I am struggling to answer is: why two implementations of chi-square return different results? $\endgroup$
    – Data Man
    Oct 24 '21 at 22:06
  • $\begingroup$ Yes, I saw that thread too (regarding your Update note). There are a few more. I am not surprised that slight differences may exist between two implementations of the same theoretical process. I saw this often in the past with other packages too. Many things, such as the nature of floating point operations may play a role and it should be accepted. But the difference I am seeing is so dramatic, and shifts the business conclusions so radically that I cannot just gloss over it. Heck, I feel I am doing something wrong, if I only knew what. $\endgroup$
    – Data Man
    Oct 25 '21 at 18:23
0
$\begingroup$

As follow-up to this thread I posted a bug report to Scikit-Learn. And yes, yesterday this issue has been confirmed as the bug in Scikit-Learn. Indeed, the results of SelectKBest are incorrect, and this may possibly be impacting a large number of ML projects worldwide. I described the details and the possible temporary remedies in this article.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.