How does sklearn.SelectKBest uses chi2 test on continous data?

Question

I am asking a question very closely related to this one (same question as one of the answers).

https://stackoverflow.com/questions/49847493/using-chi2-test-for-feature-selection-with-continuous-features-scikit-learn

My understanding is that chi2 test can only be used when features and the target are categorical variables, represented as binary. However, sklearn example has continuous data for features and target.

https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html

>>> from sklearn.datasets import load_digits
>>> from sklearn.feature_selection import SelectKBest, chi2
>>> X, y = load_digits(return_X_y=True)
>>> X.shape
(1797, 64)
>>> X_new = SelectKBest(chi2, k=20).fit_transform(X, y)
>>> X_new.shape
(1797, 20)

I also tested this on a small toy example and feature selection looks to be giving reasonable results.

What is going on here? How does this work? My first thought is that the data gets binned under the hood, but I am still not clear how that would work.

Answer 1

My understand of this is that when using Chi2 for feature selection, the dependent variable has to be categorical type, but the independent variables can be either categorical or continuous variables, as long as it's non-negative. What the algorithm trying to do is firstly build a contingency table in a matrix format that reveals the multivariate frequency distribution of the variables. Then try to find the dependence structure underlying the variables using this contingency table. The Chi2 is one way to measure the dependency.

From the Wikipedia on contingency table (https://en.wikipedia.org/wiki/Contingency_table, 2020-07-04):

Standard contents of a contingency table

• Multiple columns (historically, they were designed to use up all the white space of a printed page). Where each row refers to a specific sub-group in the population (in this case men or women), the columns are sometimes referred to as banner points or cuts (and the rows are sometimes referred to as stubs).
• Significance tests. Typically, either column comparisons, which test for differences between columns and display these results using letters, or, cell comparisons, which use color or arrows to identify a cell in a table that stands out in some way.
• Nets or netts which are sub-totals.
• One or more of: percentages, row percentages, column percentages, indexes or averages.
• Unweighted sample sizes (counts).

Based on this, pure binary features can be easily summed up as counts, which is how people conduct the Chi2 test usually. But as long as the features are non-negative, one can always accumulated it in the contingency table in a "meaningful" way.

Answer 2

I was looking at the code of that function (SelectKBest, internally using chi2) last week. The code is really intended for binary variables. Because of the nature of the computation, sometimes it may appear to give reasonable results also for categorical, or even continuous variables. But these results cannot be trusted.

In my understanding, you cannot use chi-square (chi2) for continuous variables.The chi2 calculation requires to build the contingency table, where you count occurrences of each category of the variables of interest. As the cells in that RC table correspond to particular categories, I cannot see how such table could be built from continuous variables without significant preprocessing.

But there are more problems with the existing implementation of the chi2 feature reduction in Scikit-learn. As mentioned above, the implementation requires binary features, but the documentation is not clear about this. This led to common perception in the community that SelectKBest could be used for categorical features, while in fact it cannot. Second, the Scikit-learn implementation fails to implement the chi2 condition (80% cells of RC table need to have expected count >=5) which leads to incorrect results for categorical features with many possible values. All in all, in my view this method should not be used neither for continuous, nor for categorical features. I wrote more about this below:

Here is the Scikit-learn bug request #21455: and here the article and the alternative implementation: