I am trying to learn feature importance for an ML model. To do this I am running a univariate statistical test between each independent feature and the dependent variable I am trying to predict.
The dependent variable is binary and the features are mixed (some are categorical some are continuous). As a result, for the features that are continuous, I am running a two sample $t$-test and for the categorical features I am running a $\chi^2$-test.
I want to be able to compare the outputs of these two different tests in order to see which features are the most closely related to the dependent variable. How do I compare the results from these two different tests?
I am not sure how to do this as the test statistics are obviously different and just because the test shows that the relationship is statistically significant, this of course does not mean that the relationship is a particularly important one. For example, the mean difference between Group A and Group B could be very statistically significant but only be a mean difference of a mere 0.01 units.
Therefore, how do I find the features that not only have the largest magnitude relationship with the dependent variable I am trying to predict but also the highest statistical significance.
Thanks in advance for any help and let me know if I should provide more of my particular example.
I am using scikit learn's
chi2 methods to do this analysis. (They do what I previously described. two sample $t$-test and $\chi^2$-tests, respectively.)