In terms of neural network lingo (y = Weight * x + bias) how would I know which variables are more important than others?

I have a neural network with 10 inputs, 1 hidden layer with 20 nodes, and 1 output layer which has 1 node. I'm not sure how to know which input variables are more influential than other variables. What I'm thinking is that if an input is important then it will have a highly weighted connection to the first layer, but the weight might be positive or negative. So what I might do is take the absolute value of the input's weights and sum them. The more important inputs would have higher sums.

So for example, if hair length is one of the inputs, then it should have 1 connection to each of the nodes in the next layer, so 20 connections (and therefore 20 weights). Can I just take the absolute value of each weight and sum them together?

  • 3
    $\begingroup$ You certainly can do that, but it's not clear what it really means besides "the sum of all the weights for this variable over all the connections". Calling it "importance" is entirely arbitrary. $\endgroup$ Feb 9 '17 at 23:56
  • 1
    $\begingroup$ I just want ANY information that would suggest a variable's important, and I think that this might be a good way. $\endgroup$ Feb 9 '17 at 23:59

What you describe is indeed one standard way of quantifying the importance of neural-net inputs. Note that in order for this to work, however, the input variables must be normalized in some way. Otherwise weights corresponding to input variables that tend to have larger values will be proportionally smaller. There are different normalization schemes, such as for instance subtracting off a variable's mean and dividing by its standard deviation. If the variables weren't normalized in the first place, you could perform a correction on the weights themselves in the importance calculation, such as multiplying by the standard deviation of the variable.

$I_i = \sigma_i\sum\limits_{j = 1}^{n_\text{hidden}}\left|w_{ij}\right|$.

Here $\sigma_i$ is the standard deviation of the $i$th input, $I_i$ is the $i$th input's importance, $w_{ij}$ is the weight connecting the $i$th input to the $j$th hidden node in the first layer, and $n_\text{hidden}$ is the number of hidden nodes in the first layer.

Another technique is to use the derivative of the neural-net mapping with respect to the input in question, averaged over inputs.

$I_i = \sigma_i\left\langle\left|\frac{dy}{dx_i}\right|\right\rangle$

Here $x_i$ is the $i$th input, $y$ is the output, and the expectation value is taken with respect to the vector of inputs $\mathbf{x}$.

  • $\begingroup$ Would this work if I only use the connections between the inputs and the first hidden layer (rather than use all the hidden layers)? $\endgroup$ Feb 9 '17 at 23:31
  • $\begingroup$ You should only use the first hidden layer. After one layer, the other weights are not tied to one input any more than another. I edited the answer slightly to clarify this. $\endgroup$ Feb 9 '17 at 23:34
  • $\begingroup$ I remembered another approach and added it to the answer. $\endgroup$ Feb 9 '17 at 23:43

A somewhat brute force but effective solution:

Try 'droping' an input by using a constant for one of your input features. Then, train the network for each of the possible cases and see how your accuracy drops. Important inputs will provide the greatest benefit to overall accuracy.

  • $\begingroup$ That's certainly a possibility. $\endgroup$
    – SmallChess
    Feb 10 '17 at 8:11
  • 9
    $\begingroup$ There's a pitfall though: even if a feature is extremely important, if there's another feature highly correlated to the first, neither will be considered important by your criteria (the lack of the first is compensated by the presence of the latter, while less informative but more 'unique' features will appear more important that way) $\endgroup$
    – Firebug
    Dec 15 '18 at 23:10
  • $\begingroup$ This is sometime called ablation (more specifically micro ablation) testing $\endgroup$ Dec 16 '18 at 13:54

What you described is not "deep network", where you only have $10$ inputs and $5$ units in hidden layer. When people say deep learning, it usually means hundreds of thousands of hidden units.

For a shallow network, this gives an example of define the variable importance.

For a really deep network, people do not talk about variable importance too much. Because the inputs are raw level features, such as pixels in an image.

  • $\begingroup$ I edited my comment to reflect what I meant. I meant to say 20 nodes in the first layer, not 5 nodes. Great share and thanks for distinguishing shallow/deep nets. $\endgroup$ Feb 10 '17 at 4:08
  • 10
    $\begingroup$ @hxd1011 not to be pedantic prude, but deep means more layers not thousands of hidden units :). $\endgroup$
    – Rafael
    Feb 10 '17 at 4:23

The most that ive found about this is elaborately listed on this site more specifically you can look at this. If you talk only about linear models then you have to normalize the weights to make them interpret-able but even this can be misleading more on this on the link mentioned . Some people tried making complex functions of weights to interpret importance's of inputs (Garson's , Gedeon's and Milne's ) but even this can be misleading you can find more about this once you scroll the first link i mentioned. In general i would advice to go ahead interpret the results with a grain of salt.

would agree with @rhadar's answer but would like to add that instead of using any constant try using the mean value for that input and don't forget to retrain the network.

PS: sorry could not post more links or comment here don't have much reputation.


Given that you have:

  1. A classification task
  2. A trained model
  3. Normalised features (between 0 and 1)

Has anyone tried:

  1. Zeroing out the biases
  2. Pass each time as features a one hot vector where all features are zero except one.
  3. Examine the output.

In that case, I think the output would be a number designating the "importance" of the feature as this output would also represent the output of the path of this 1 signal inside the network.

It is like lighting only one lightbulb inside a labyrinth and measure the light coming out in the exit.

  • $\begingroup$ I don't think that this would help much. What you would like assessing is how much the variation of that input would affect the output (by itself or in combination with other features) $\endgroup$
    – elachell
    Nov 7 '18 at 17:02

You can also compute permutation importance of the input variables: https://scikit-learn.org/stable/modules/permutation_importance.html

It is model-agnostic and is applicable to measure importance of input variables for “black-box” models like neural networks.

  • $\begingroup$ This answer could be made more useful by expanding upon the method and its appropriate use in addition to the link. Quoting or referencing the most important part of the link would be useful too. See here for guidance on answering effectively: stats.stackexchange.com/help/how-to-answer "...Always quote the most relevant part of an important link, in case the external resource is unreachable or goes permanently offline." $\endgroup$
    – dlid
    Apr 30 at 3:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.