What are the differences between a Chi-squared test of independence and sklearn feature_importances_? If I use a Chi-squared test to determine the independence of one of the categorical features from my dataset against the label set, and the test determines statistical significance, i.e. the label is indeed dependent, can I confidently assume that when I run the feature_importances_ function from sklearn in python, that I should also expect high results from this particular feature?
Also, I understand that the Chi-squared test of independent can only test for independence between two sets of categorical variables. That being said, which statistical hypothesis test can test for independence between a non-categorical variable and the label set (which may or may not be categorical)?
 A: In regression, the relationship between the feature and the label is examined by t-statistic. The null hypothesis is that there is no relationship and hence the coefficient is 0. If the data suggests otherwise, when the p-value is less than a threshold (e.g. 0.05), you reject the null. This applies no matter if your feature is categorical or continuous. Although when you have categorical feature and have done one-hot-encoding, then you will need to use partial F-statistics, because you have to test for all one-hot-encoded columns of a single feature simultaneously.
The feature importance from sklearn is something very different. If you have a classification problem, it measures the average reduction in impurity (basically how much does one feature/node contribute in successfully splitting the dataset into purer subgroups). If you have a regression problem, it measures the average reduction in variance.
You can find some further explanations t-statistics in the regression setting on this website. Or if you search on StackExchange, you might find many similar posts talking about this topic.
