TLDR: From my understanding, as Chi-Square is a statistic dependent on the degrees of freedom, it cannot be used to compare reliably across different features, but it seems to be used in Sklearn for example. In my view, p-values should be used instead.
I am juggling my mind over the following:
- The p-value for a given Chi-Square statistic depends on the degrees of freedom (namely
(columns - 1) * (rows - 1)).
- This means that, for a given value of the Chi-Square statistic (say 2), it might or might not translate into statistical significance (say at
alpha = 0.5), depending on the degrees of freedom.
- When applied to different features against the target, this could result in different degrees of freedom.
- To better illustrate, let's say the target is binary, and feature 1 is categorical with 3 distinct values, while feature 2 is also categorical with 10 potential values. The degrees of freedom for feature 1 would be 2, and for feature 2 would be 9.
- When I look into Sklearn's
chi2code and documentation, I conclude that the Chi-Square statistic is in fact used to sort the features for subsequent selection.
- This could result in situations in which we are leaving in some features that are not significantly related to the target (as per the statistical test) while dropping others that are significantly related to the target.
- Using p-values instead would solve this issue, as these are already adjusted for the degrees of freedom.
Could you help me by identifying where my reasoning is failing (if it is indeed)?