1
$\begingroup$

If I use a Chi-squared test to determine the independence of one of the categorical features from my dataset against the label set, and the test determines statistical significance, i.e. the label is indeed dependent, can I confidently assume that when I run the feature_importances_ function from sklearn in python, that I should also expect high results from this particular feature?

Also, I understand that the Chi-squared test of independent can only test for independence between two sets of categorical variables. That being said, which statistical hypothesis test can test for independence between a non-categorical variable and the label set (which may or may not be categorical)?

$\endgroup$
4
  • $\begingroup$ Could you explain what feature_importances_ is supposed to do? In particular, is it based on a multiple regression or just the univariate regression of these particular two variables? (2) What you understand is not correct. There are versions of the chi-squared test for any number of variables. (3) Almost any form of regression model can be made to yield tests of independence (or at least lack of the kind of association that they do model). Most of them conduct such a test by default. $\endgroup$
    – whuber
    Commented Dec 22, 2021 at 23:08
  • $\begingroup$ From the documentation, The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance. I would assume it is based on the strength of the r-value in the multivariate regression of all the features without the feature in question. $\endgroup$ Commented Dec 22, 2021 at 23:19
  • $\begingroup$ Addressing your other points, then if i am trying to determine the independence between a feature which is continuous, and the label set which is categorical, then how can I use Chi-squared test to determine if the continuous feature is dependent or not? $\endgroup$ Commented Dec 22, 2021 at 23:21
  • 1
    $\begingroup$ When one or more features are continuous, do a regression. Independence implies the regression will not be significantly better than fitting the grand mean. $\endgroup$
    – whuber
    Commented Dec 22, 2021 at 23:32

1 Answer 1

2
$\begingroup$

In regression, the relationship between the feature and the label is examined by t-statistic. The null hypothesis is that there is no relationship and hence the coefficient is 0. If the data suggests otherwise, when the p-value is less than a threshold (e.g. 0.05), you reject the null. This applies no matter if your feature is categorical or continuous. Although when you have categorical feature and have done one-hot-encoding, then you will need to use partial F-statistics, because you have to test for all one-hot-encoded columns of a single feature simultaneously.

The feature importance from sklearn is something very different. If you have a classification problem, it measures the average reduction in impurity (basically how much does one feature/node contribute in successfully splitting the dataset into purer subgroups). If you have a regression problem, it measures the average reduction in variance.

You can find some further explanations t-statistics in the regression setting on this website. Or if you search on StackExchange, you might find many similar posts talking about this topic.

$\endgroup$
2
  • $\begingroup$ Thank you. In other words, the feature importance function in sklearn computes the reduction in the r value when the feature is removed? $\endgroup$ Commented Dec 22, 2021 at 23:48
  • $\begingroup$ The feature might appear multiple times in different nodes, and every time it appears, there is some contribution it makes. It is the average contribution of this feature to the reduction in impurity/or variance. $\endgroup$
    – user344849
    Commented Dec 23, 2021 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.