# How to compare $t$-tests to $\chi^2$-tests for feature selection

I am trying to learn feature importance for an ML model. To do this I am running a univariate statistical test between each independent feature and the dependent variable I am trying to predict.

The dependent variable is binary and the features are mixed (some are categorical some are continuous). As a result, for the features that are continuous, I am running a two sample $t$-test and for the categorical features I am running a $\chi^2$-test.

I want to be able to compare the outputs of these two different tests in order to see which features are the most closely related to the dependent variable. How do I compare the results from these two different tests?

I am not sure how to do this as the test statistics are obviously different and just because the test shows that the relationship is statistically significant, this of course does not mean that the relationship is a particularly important one. For example, the mean difference between Group A and Group B could be very statistically significant but only be a mean difference of a mere 0.01 units.

Therefore, how do I find the features that not only have the largest magnitude relationship with the dependent variable I am trying to predict but also the highest statistical significance.

Thanks in advance for any help and let me know if I should provide more of my particular example.

I am using scikit learn's feature_selection f_classif, chi2 methods to do this analysis. (They do what I previously described. two sample $t$-test and $\chi^2$-tests, respectively.)

• As you'll find on most of the highest rated answers here, this kind of feature selection based on statistical significance testing is deferred to e.g. shrinkage models (lasso, ridge) or exclusion of variables based on expert knowledge. Any kind of statistical test for predictive power will essentially result in the data being used twice (to select features and to predict the outcome with them). This will almost always result in serious overfitting and biased results. – Frans Rodenburg Jul 27 '18 at 5:20
• Thanks Frans. You mean that because I would be running so many univariate tests there is a high probability that this would create many false positives? Moreover, if this is the case then can I not use the Bonferroni correction? – tbrick Jul 27 '18 at 15:49
• Unfortunately, this is a fundamental issue that is not solved by multiple testing correction. Significance tests are not appropriate for finding predictors to include in the model. – Frans Rodenburg Jul 27 '18 at 19:57
• Can you link me to a discussion of why this is the case? – tbrick Jul 28 '18 at 17:31
• This is the best explanation on here, in my opinion: stats.stackexchange.com/a/20856/176202 – Frans Rodenburg Jul 29 '18 at 3:02

2. The $t$-test compares the means of the groups, in relation to their standard error. The $\chi^2$-test compares the observed proportions of categories in either group to their expected proportions (expected if there were no difference). The $p$-values of these tests are evidence against different null-hypotheses.