Given a set of $n$ instances. For each instance I have a feature vector consisting of $m$ (numerical) features ($x_1$, $x_2$,...,$x_m$), n>>m. Moreover, for each instance I have a numerical score $y$ (observable). I would like to:

  1. find out which subset of features, or linear combination thereof, explains the scores the best.
  2. create a nice visualization for this.

I have been pointed to Principal Component Analysis (PCA). The problem with PCA is that it only takes the feature vectors into account; PCA does not relate the features to the numerical score $y$.

Practical application: Given a large number of problem instances (e.g. traveling salesman problems) and some algorithm to solve the problem. Each time we solve the instance we can measure the total time (=score) it took to solve the instance. Moreover, for each instance we can obtain a number of features, e.g. size of te instance, graph diameter, etc. Which of these features explain the computation time best?

  • $\begingroup$ By "instances" do you mean observations (i.e. usually "rows" of data)? $\endgroup$ – StatsStudent Nov 24 '18 at 4:14
  • $\begingroup$ Can you not use a regression modelling setting and then perform some form of model selection? I guess what type of modelling you use will depend on whether you can assume linearity or nonlinearity of the relationship between each feature and y controlling for the remaining features. For linearity, linear regression modelling would do. For nonlinearity, generalized additive modelling could be an option. $\endgroup$ – Isabella Ghement Nov 24 '18 at 4:15
  • $\begingroup$ Presuming you have fewer predictors than instances (i.e., m < n), you could perform model selection based on the BIC criterion, which is suitable in explanatory settings. As far as the visualization of the final model goes, effect plots could work - they would enable you to display the effect of each feature on y after controlling for the remaing features included in the final model. See socialsciences.mcmaster.ca/jfox/Papers/…. $\endgroup$ – Isabella Ghement Nov 24 '18 at 4:23
  • $\begingroup$ Could you clarify the type of relationship between the scores and input features (e.g. is it linear)? It's hard to tell from the phrasing in (1) (e.g. you might be considering scores to be a nonlinear function of a linear combination of features). Also, do you have a particular metric in mind for "best explaining the scores'? $\endgroup$ – user20160 Nov 24 '18 at 12:34
  • $\begingroup$ I think you need a feature selection algorithm: en.wikipedia.org/wiki/Feature_selection $\endgroup$ – Steve Prestwich Nov 25 '18 at 15:37

There is a lot of options, it depends what exactly do you want.

Feature importance or permutation importance

Both methods tells you which features are most important for the model. It is a number for each feature. It is calculated after the model is fitted. It doesn't tell you anything about which values of a feature implies what scores.

In sklearn most modelz has model.feature_importances_. Sum of all feature importances is 1.

Permutation importance is calculated for a fitted model. It tells you how much the metric worsens if you shuffle the feature column.


    base_score = model.score(x_dev, y_dev)
    for i in range(nr_features):
        x_dev_copy = copy(x_dev)
        x_dev_copy[:, i] = shuffle(x_dev_copy[:, i])
        perm_score = model.score(x_dev_copy, y_dev)
        perm_imp[i] = (perm_score - base_score) / base_score

You can read more about permutation importance here.

Partial Dependence Plots

tells you what values of a feature increases/decreases the values of prediction. It looks like this:

enter image description here

More info on Kaggle: Partial Dependence Plots or go straight to the library PDPbox GitHub.

SHAP value

explains why the model gives particular prediction for given instance. It plots the following graph which tells you which feature values moved the prediction from an average value to current value for the current instance.

enter image description here

Check SHAP library for more details.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.