1
$\begingroup$

I am asking a question very closely related to this one (same question as one of the answers).

https://stackoverflow.com/questions/49847493/using-chi2-test-for-feature-selection-with-continuous-features-scikit-learn

My understanding is that chi2 test can only be used when features and the target are categorical variables, represented as binary. However, sklearn example has continuous data for features and target.

https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html

>>> from sklearn.datasets import load_digits
>>> from sklearn.feature_selection import SelectKBest, chi2
>>> X, y = load_digits(return_X_y=True)
>>> X.shape
(1797, 64)
>>> X_new = SelectKBest(chi2, k=20).fit_transform(X, y)
>>> X_new.shape
(1797, 20)

I also tested this on a small toy example and feature selection looks to be giving reasonable results.

What is going on here? How does this work? My first thought is that the data gets binned under the hood, but I am still not clear how that would work.

$\endgroup$
  • $\begingroup$ The source code shows that the chi squared can be only applied to non negative features. These are converted to frequencies. Digits pixels are considered Bernoulli distributed, with each intensity equal to the probability of being 1. $\endgroup$ – user289381 Jul 4 at 0:52
1
$\begingroup$

My understand of this is that when using Chi2 for feature selection, the dependent variable has to be categorical type, but the independent variables can be either categorical or continuous variables, as long as it's non-negative. What the algorithm trying to do is firstly build a contingency table in a matrix format that reveals the multivariate frequency distribution of the variables. Then try to find the dependence structure underlying the variables using this contingency table. The Chi2 is one way to measure the dependency.

From the Wikipedia on contingency table (https://en.wikipedia.org/wiki/Contingency_table, 2020-07-04):

Standard contents of a contingency table

  • Multiple columns (historically, they were designed to use up all the white space of a printed page). Where each row refers to a specific sub-group in the population (in this case men or women), the columns are sometimes referred to as banner points or cuts (and the rows are sometimes referred to as stubs).
  • Significance tests. Typically, either column comparisons, which test for differences between columns and display these results using letters, or, cell comparisons, which use color or arrows to identify a cell in a table that stands out in some way.
  • Nets or netts which are sub-totals.
  • One or more of: percentages, row percentages, column percentages, indexes or averages.
  • Unweighted sample sizes (counts).

Based on this, pure binary features can be easily summed up as counts, which is how people conduct the Chi2 test usually. But as long as the features are non-negative, one can always accumulated it in the contingency table in a "meaningful" way.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.