What is the industry standard way for determining feature “importance”?

Question

My understanding is, there are different tests to run such as ANOVA, Pearson's Correlation, Chi-Square. Choosing these tests is dependent on if the features / responses are categorial / continuous. And then each test has it's own test specific ways to measure "importance", for example: https://chrisalbon.com/machine_learning/feature_selection/anova_f-value_for_feature_selection/

The MAIN test that I've seen though is Random Forest. I've used Random Forest's to get feature importance graphs like this: https://imgur.com/a/FY3muq1.

However, I'm not really sure what is like the high-level or industry standard way to respond when someone asks something like "how would you determine the features that strongly affect whether or not someone buys a product?". In the past I've defaulted to discussing the RF feature importance projects I've done, but

1) I'm not super confident regarding the math behind RF feature importance

2) defaulting to RF seems super specific and might make it seem like I don't understand feature importance in general

3) I don't know what to show as a final result other than just the feature importance graph. Like for example, I don't know how to get really quantitative and make a claim like "Feature A is 2x more influential / important than Feature B".

Answer 1

I do not think there is an industry standard in any industry. Feature importance is extremely domain and application specific.

Since you started with Random Forest (RF) feature importance, note that RF importance is always measured in terms of some other metric. As in: how much does out-of-bag quality (in terms of that other metric, typically Gini, node impurity, or MSE) deteriorate if we randomly permute one feature? This is interpretable to the degree that the original metric is interpretable. The people you are talking to may be able to understand the MSE, but you may draw a total blank if you discuss Gini coefficients, which are not all that intuitive to start with.

Which leads me to a recommendation. As above, there is really no accepted measure of "importance". So you are free to (and should) calculate something that is understandable for your audience.

In the example you cite, about features influencing whether or not someone buys a product, one way of going about this might be looking at fitted or predicted probabilities. If a dog owner has a 10% conversion rate and a non-dog owner only 5%, then owning a dog increases the conversion probability by 5%. If the corresponding figures for cats are only 1%, then "owning a dog" is a more important feature than "owning a cat". This is simple, and it can be communicated to pretty much anyone.

It would be somewhat more advanced to assess whether including a feature improves predictive performance. One problem here is that assessing predictive performance is somewhat harder than it looks.

Either approach can be taken on a micro level, as above, or on an aggregated one ("which promotion leads to higher sales in the aggregate?").

And in either approach, interactions between features can be problematic. For instance, in supermarket sales forecasting, promotions always include both a price reduction and a communication to the customer, e.g., a notification in the app. Both come together, so they are highly correlated. It will be very hard to assess the importance of either the price reduction or the communication separately. (But an algorithm will happily output a number. So think carefully about whether the number is meaningful.) Ulrike Grömping has done some work to address this, particularly in the context of RFs and linear regression.