0
$\begingroup$

When looking for correlation between features (for feature selection), I found that sklearn implementation of Chi2 test of independence produce significantly different results from scipy.stats implementation.

My data contains 300 records, with 6 anonymized categorical features and the label. My focus is on the feature A. This data is available here in github. In the folder, see the file sample300.csv, alongside the notebook with my testing code.

For the feature A, sklearn's SelectKBest() returned the lowest ranking, suggesting there is no correlation between A and label. But scipy.stats.chi2_contingency() returned very different result, suggesting the correlation is very high.

Because of mismatch between the two, I went a long way performing a number of different tests described in detail in this article The results suggest that the scipy implementation is correct, while sklearn implementation is incorrect.

This conclusion surprising, given the popularity of sklearn.feature_selection_SelectKBest. Why are the two implementations of chi2 producing contradicting results? Did I make a methodical error somewhere or is this really a bug? I am looking forward for some critical peer review.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

The bug report mentioned in the linked article has been since closed by this commit. This wasn't a case of faulty implementation, but at worst a case of unclear documentation.

Back then, the SelectKBest function was a wrapper around the sklearn chi2 function, which does not conduct a chi-squared test of independence at all, but a sort of chi-squared goodness-of-fit test.

The purpose of the sklearn chi2 function seems to be a common source of confusion, which has been adressed in this other answer: Pearson chi2 tests of independence: differences between Scipy and Scikit-learn

While there is no inconsistency or error in the Sklearn's chi2 function once you understand how it works, it has still a quite idiosyncratic purpose and syntax. I'd recommend using scipy instead to anyone looking for something simple in Python. From my experience, their implementations of chi-squared tests are perfectly reliable, and consistent with other statistical software: chi-squared goodness-of-fit test and chi-squared test of independence.

Improve this answer
$\endgroup$
0
$\begingroup$

You should go back to the literature on Chi2 tests and about what they can tell you, and in this case what they cannot tell you. It's a reasoning issue in your case.

Indeed, the test will tell you that there is a strong correlation between experience and success. But that it doesn't tell you there isn't one between education and succes doesn't necessarily mean there isn't one. You are working under the assumption that education and experience are the only determinant factors of success, which is wrong. You are not controlling for all other factors that are predictors of success, some of which might not be observable (e.g., personality traits such as conscienciousness). Many of them are strongly correlated to education and as long as you are not taking them into account, you can say nothing about the importance of education in success.

In order words, you are using this python-package and the test it uses in the wrong way.

Update after comment : But scipy and sklearn implement chi2 in slightly different ways to begin with. This is a known difference between the two methods. However, this does not give huge differences between the results. (https://stackoverflow.com/questions/50932433/scipy-and-sklearn-chi2-implementations-give-different-results).

Whether A and B are education and experience or not, you should still be careful to control for all variables.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I appreciate this answer. This tells me clearly that I might have misled some readers so I want to add this clarification: The original meaning of feature A does not matter. In the article (meant for general public) I called this feature 'education' to build a story. But here let's just call it feature A and forget about its business meaning. The meaning does not matter. The question I am struggling to answer is: why two implementations of chi-square return different results? $\endgroup$
    – Data Man
    Oct 24, 2021 at 22:06
  • $\begingroup$ Yes, I saw that thread too (regarding your Update note). There are a few more. I am not surprised that slight differences may exist between two implementations of the same theoretical process. I saw this often in the past with other packages too. Many things, such as the nature of floating point operations may play a role and it should be accepted. But the difference I am seeing is so dramatic, and shifts the business conclusions so radically that I cannot just gloss over it. Heck, I feel I am doing something wrong, if I only knew what. $\endgroup$
    – Data Man
    Oct 25, 2021 at 18:23
0
$\begingroup$

As follow-up to this thread I posted a bug report to Scikit-Learn. And yes, yesterday this issue has been confirmed as the bug in Scikit-Learn. Indeed, the results of SelectKBest are incorrect, and this may possibly be impacting a large number of ML projects worldwide. I described the details and the possible temporary remedies in this article.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.