Feature selection using chi squared for continuous features

Question

I'm looking at univariate feature selection. A method that is often described, is to look at the p-values for a $\chi^2$-test. However, I'm confused as to how this works for continuous variables.

1. How can the $\chi^2$-test work for feature selection for continuous variables? I have always thought this test works for counts. It appears to me you have to bin the data in some way, but the outcomes are dependent on the binning you choose. I'm also interested in how this works for a combination of continuous and categorical variables.

2. Is it a problem that this test is scale dependent? My second concern is that the test is scale dependent. This is not a problem for counts, which have no units of measurement, but it can have great impact on feature selection for continuous variables that are measured in some units of measurement (see Example).

Example

Showing the test is scale-dependent for variables with units of measurement:

Let's look at the original example from: http://scikit-learn.org/stable/modules/feature_selection.html

from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
iris = load_iris()
X, y = iris.data, iris.target
selector = SelectKBest(chi2, k=2)
selector.fit(X, y)
print(selector.pvalues_)
print(selector.get_support())

Output:

[False False True True]
[4.47e-03 1.657e-01 5.94e-26 2.50e-15]

Now let's imagine we had recorded the first and third column not in cm, but in mm. Obviously, this doesn't change the dependence of the class type on sepal and petal length. However, the p-values change strongly, and accordingly, the selected columns change:

X[:, 0] = 10*X[:, 0]
X[:, 2] = 10*X[:, 2]
selector.fit(X, y)
print(selector.pvalues_)
print(selector.get_support())

Output

[True False True False] 
[3.23e-024 1.66e-001 5.50e-253 2.50e-015]

If I had also recorded the 2nd column in mm instead of cm, that would also have given me a significant p-value.

I believe this had to with the fact that the method does not implement any binning but sums all values and compares that to the expected sum. Additionally, I believe the fact that the numerator in the $\chi^2$ is squared while the denominator is not adds to the problem.

Answer 1

I think you confuse the data itself (which can be continuous) with the fact that when you talk about data, you actually talk about samples, which are discrete.

The $\chi^2$ test (in wikipedia and the model selection by $\chi^2$ criterion) is a test to check for independence of sampled data. I.e. when you have two (or more) of sources of the data (i.e. different features), and you want to select only features that are mutually independent, you can test it by rejecting the Null hypothesis (i.e. data samples are dependant) if the probability to encounter such a sample (under the Null hypothesis), i.e. the p value, is smaller than some threshold value (e.g., p < 0.05).

So now for your questions,

1. The $\chi^2$ test do work only on categorical data, as you must count the occurences of the samples in each category to use it, but as I've mentioned above, when you use it, you actually have samples in hand, so one thing you can do is to divide your samples into categories based on thresholds (e.g., $cat_1: x \in [th_1 < x < th_2], cat_2: x \in [th_2 < x < th_3]$, etc.) and count all the samples that fall into each category.
2. As for the scales - you are obviously must use the same scales when you discretize your samples, otherwise it won't make any sense, but when you conduct the $\chi^2$ test itself, as you've correctly pointed out, you are dealing with counts, so they won't have any scales anyway.

Cheers.