Deep learning : How do I know which variables are important?

Question

In terms of neural network lingo (y = Weight * x + bias) how would I know which variables are more important than others?

I have a neural network with 10 inputs, 1 hidden layer with 20 nodes, and 1 output layer which has 1 node. I'm not sure how to know which input variables are more influential than other variables. What I'm thinking is that if an input is important then it will have a highly weighted connection to the first layer, but the weight might be positive or negative. So what I might do is take the absolute value of the input's weights and sum them. The more important inputs would have higher sums.

So for example, if hair length is one of the inputs, then it should have 1 connection to each of the nodes in the next layer, so 20 connections (and therefore 20 weights). Can I just take the absolute value of each weight and sum them together?

Answer 1

What you describe is indeed one standard way of quantifying the importance of neural-net inputs. Note that in order for this to work, however, the input variables must be normalized in some way. Otherwise weights corresponding to input variables that tend to have larger values will be proportionally smaller. There are different normalization schemes, such as for instance subtracting off a variable's mean and dividing by its standard deviation. If the variables weren't normalized in the first place, you could perform a correction on the weights themselves in the importance calculation, such as multiplying by the standard deviation of the variable.

$I_i = \sigma_i\sum\limits_{j = 1}^{n_\text{hidden}}\left|w_{ij}\right|$.

Here $\sigma_i$ is the standard deviation of the $i$th input, $I_i$ is the $i$th input's importance, $w_{ij}$ is the weight connecting the $i$th input to the $j$th hidden node in the first layer, and $n_\text{hidden}$ is the number of hidden nodes in the first layer.

Another technique is to use the derivative of the neural-net mapping with respect to the input in question, averaged over inputs.

$I_i = \sigma_i\left\langle\left|\frac{dy}{dx_i}\right|\right\rangle$

Here $x_i$ is the $i$th input, $y$ is the output, and the expectation value is taken with respect to the vector of inputs $\mathbf{x}$.

Answer 2

A somewhat brute force but effective solution:

Try 'droping' an input by using a constant for one of your input features. Then, train the network for each of the possible cases and see how your accuracy drops. Important inputs will provide the greatest benefit to overall accuracy.

Answer 3

What you described is not "deep network", where you only have $10$ inputs and $5$ units in hidden layer. When people say deep learning, it usually means hundreds of thousands of hidden units.

For a shallow network, this gives an example of define the variable importance.

For a really deep network, people do not talk about variable importance too much. Because the inputs are raw level features, such as pixels in an image.

Answer 4

The most that ive found about this is elaborately listed on this site more specifically you can look at this. If you talk only about linear models then you have to normalize the weights to make them interpret-able but even this can be misleading more on this on the link mentioned . Some people tried making complex functions of weights to interpret importance's of inputs (Garson's , Gedeon's and Milne's ) but even this can be misleading you can find more about this once you scroll the first link i mentioned. In general i would advice to go ahead interpret the results with a grain of salt.

would agree with @rhadar's answer but would like to add that instead of using any constant try using the mean value for that input and don't forget to retrain the network.

PS: sorry could not post more links or comment here don't have much reputation.

Answer 5

Given that you have:

1. A classification task
2. A trained model
3. Normalised features (between 0 and 1)

Has anyone tried:

1. Zeroing out the biases
2. Pass each time as features a one hot vector where all features are zero except one.
3. Examine the output.

In that case, I think the output would be a number designating the "importance" of the feature as this output would also represent the output of the path of this 1 signal inside the network.

It is like lighting only one lightbulb inside a labyrinth and measure the light coming out in the exit.

Answer 6

You can also compute permutation importance of the input variables: https://scikit-learn.org/stable/modules/permutation_importance.html

It is model-agnostic and is applicable to measure importance of input variables for “black-box” models like neural networks.

Answer 7

What you describe is IMHO a simple and effective way to determine what inputs your model is most sensitive to.

However, 'sensitive' is not necessarily the same as 'important'.

For example if your model is very prone to overfitting issues then such a metric could easily lead you in the wrong direction: Those 'highly sensitive inputs' could then actually just mean 'highly important for easy overfitting'.