inferring most important features Given a set of $n$ instances. For each instance I have a feature vector consisting of $m$ (numerical) features ($x_1$, $x_2$,...,$x_m$), n>>m. Moreover, for each instance I have a numerical score $y$ (observable). I would like to:


*

*find out which subset of features, or linear combination thereof, explains the scores the best.

*create a nice visualization for this.


I have been pointed to Principal Component Analysis (PCA). The problem with PCA is that it only takes the feature vectors into account; PCA does not relate the features to the numerical score $y$.
Practical application:
Given a large number of problem instances (e.g. traveling salesman problems) and some algorithm to solve the problem. Each time we solve the instance we can measure the total time (=score) it took to solve the instance. Moreover, for each instance we can obtain a number of features, e.g. size of te instance, graph diameter, etc. Which of these features explain the computation time best?
 A: There is a lot of options, it depends what exactly do you want.
Feature importance or permutation importance
Both methods tells you which features are most important for the model. It is a number for each feature. It is calculated after the model is fitted. It doesn't tell you anything about which values of a feature implies what scores.
In sklearn most modelz has model.feature_importances_. Sum of all feature importances is 1.
Permutation importance is calculated for a fitted model. It tells you how much the metric  worsens if you shuffle the feature column.
Pseudo-code:
    model.fit()
    base_score = model.score(x_dev, y_dev)
    for i in range(nr_features):
        x_dev_copy = copy(x_dev)
        x_dev_copy[:, i] = shuffle(x_dev_copy[:, i])
        perm_score = model.score(x_dev_copy, y_dev)
        perm_imp[i] = (perm_score - base_score) / base_score

You can read more about permutation importance here.
Partial Dependence Plots
tells you what values of a feature increases/decreases the values of prediction. It looks like this:

More info on Kaggle: Partial Dependence Plots or go straight to the library PDPbox GitHub.
SHAP value
explains why the model gives particular prediction for given instance. It plots the following graph which tells you which feature values moved the prediction from an average value to current value for the current instance.

Check SHAP library for more details.
