30
$\begingroup$

In terms of neural network lingo (y = Weight * x + bias) how would I know which variables are more important than others?

I have a neural network with 10 inputs, 1 hidden layer with 20 nodes, and 1 output layer which has 1 node. I'm not sure how to know which input variables are more influential than other variables. What I'm thinking is that if an input is important then it will have a highly weighted connection to the first layer, but the weight might be positive or negative. So what I might do is take the absolute value of the input's weights and sum them. The more important inputs would have higher sums.

So for example, if hair length is one of the inputs, then it should have 1 connection to each of the nodes in the next layer, so 20 connections (and therefore 20 weights). Can I just take the absolute value of each weight and sum them together?

$\endgroup$
2
  • 3
    $\begingroup$ You certainly can do that, but it's not clear what it really means besides "the sum of all the weights for this variable over all the connections". Calling it "importance" is entirely arbitrary. $\endgroup$ Feb 9, 2017 at 23:56
  • 1
    $\begingroup$ I just want ANY information that would suggest a variable's important, and I think that this might be a good way. $\endgroup$ Feb 9, 2017 at 23:59

7 Answers 7

19
$\begingroup$

What you describe is indeed one standard way of quantifying the importance of neural-net inputs. Note that in order for this to work, however, the input variables must be normalized in some way. Otherwise weights corresponding to input variables that tend to have larger values will be proportionally smaller. There are different normalization schemes, such as for instance subtracting off a variable's mean and dividing by its standard deviation. If the variables weren't normalized in the first place, you could perform a correction on the weights themselves in the importance calculation, such as multiplying by the standard deviation of the variable.

$I_i = \sigma_i\sum\limits_{j = 1}^{n_\text{hidden}}\left|w_{ij}\right|$.

Here $\sigma_i$ is the standard deviation of the $i$th input, $I_i$ is the $i$th input's importance, $w_{ij}$ is the weight connecting the $i$th input to the $j$th hidden node in the first layer, and $n_\text{hidden}$ is the number of hidden nodes in the first layer.

Another technique is to use the derivative of the neural-net mapping with respect to the input in question, averaged over inputs.

$I_i = \sigma_i\left\langle\left|\frac{dy}{dx_i}\right|\right\rangle$

Here $x_i$ is the $i$th input, $y$ is the output, and the expectation value is taken with respect to the vector of inputs $\mathbf{x}$.

$\endgroup$
6
  • $\begingroup$ Would this work if I only use the connections between the inputs and the first hidden layer (rather than use all the hidden layers)? $\endgroup$ Feb 9, 2017 at 23:31
  • $\begingroup$ You should only use the first hidden layer. After one layer, the other weights are not tied to one input any more than another. I edited the answer slightly to clarify this. $\endgroup$ Feb 9, 2017 at 23:34
  • $\begingroup$ I remembered another approach and added it to the answer. $\endgroup$ Feb 9, 2017 at 23:43
  • 1
    $\begingroup$ Unfortunately I don't really have a citation for it. This was a technique I used once at the suggestion of some collaborators, but I don't think we published the feature-importance analysis anywhere. $\endgroup$ Mar 12 at 0:18
  • 1
    $\begingroup$ The gist of the technique though is that the expectation value is taken over the actual probability distribution of the inputs. So, if you have a representative set of examples, which are tuples of values of the input variables, then you iterate over those examples and for each one numerically compute the derivative of your neural network at the location in input-variable space corresponding to that example. And then you take the average of the absolute values of those derivatives and multiply by the standard deviation of the input variable, also computed using the examples. $\endgroup$ Mar 12 at 0:24
13
$\begingroup$

A somewhat brute force but effective solution:

Try 'droping' an input by using a constant for one of your input features. Then, train the network for each of the possible cases and see how your accuracy drops. Important inputs will provide the greatest benefit to overall accuracy.

$\endgroup$
5
  • $\begingroup$ That's certainly a possibility. $\endgroup$
    – ABCD
    Feb 10, 2017 at 8:11
  • 11
    $\begingroup$ There's a pitfall though: even if a feature is extremely important, if there's another feature highly correlated to the first, neither will be considered important by your criteria (the lack of the first is compensated by the presence of the latter, while less informative but more 'unique' features will appear more important that way) $\endgroup$
    – Firebug
    Dec 15, 2018 at 23:10
  • $\begingroup$ This is sometime called ablation (more specifically micro ablation) testing $\endgroup$ Dec 16, 2018 at 13:54
  • $\begingroup$ @Firebug Outside of drawing causal inferences, if you can drop a variable and not affect model performance, I am inclined to view that variable as not being so important. Do you have a different take on this? $\endgroup$
    – Dave
    Jun 2 at 19:26
  • $\begingroup$ @dave depends on what you define as important. If you're interested in knowing if any single variable would be important, in isolation, to get the same performance, then yes, I'd conclude this variable (which could even be the most predictive one) to not be important. But if you use it to describe the set of variables that gets you the same performance, which I believe is more common, then no $\endgroup$
    – Firebug
    Jun 4 at 11:01
8
$\begingroup$

What you described is not "deep network", where you only have $10$ inputs and $5$ units in hidden layer. When people say deep learning, it usually means hundreds of thousands of hidden units.

For a shallow network, this gives an example of define the variable importance.

For a really deep network, people do not talk about variable importance too much. Because the inputs are raw level features, such as pixels in an image.

$\endgroup$
2
  • $\begingroup$ I edited my comment to reflect what I meant. I meant to say 20 nodes in the first layer, not 5 nodes. Great share and thanks for distinguishing shallow/deep nets. $\endgroup$ Feb 10, 2017 at 4:08
  • 10
    $\begingroup$ @hxd1011 not to be pedantic prude, but deep means more layers not thousands of hidden units :). $\endgroup$
    – Rafael
    Feb 10, 2017 at 4:23
4
$\begingroup$

The most that ive found about this is elaborately listed on this site more specifically you can look at this. If you talk only about linear models then you have to normalize the weights to make them interpret-able but even this can be misleading more on this on the link mentioned . Some people tried making complex functions of weights to interpret importance's of inputs (Garson's , Gedeon's and Milne's ) but even this can be misleading you can find more about this once you scroll the first link i mentioned. In general i would advice to go ahead interpret the results with a grain of salt.

would agree with @rhadar's answer but would like to add that instead of using any constant try using the mean value for that input and don't forget to retrain the network.

PS: sorry could not post more links or comment here don't have much reputation.

$\endgroup$
0
1
$\begingroup$

Given that you have:

  1. A classification task
  2. A trained model
  3. Normalised features (between 0 and 1)

Has anyone tried:

  1. Zeroing out the biases
  2. Pass each time as features a one hot vector where all features are zero except one.
  3. Examine the output.

In that case, I think the output would be a number designating the "importance" of the feature as this output would also represent the output of the path of this 1 signal inside the network.

It is like lighting only one lightbulb inside a labyrinth and measure the light coming out in the exit.

$\endgroup$
1
  • $\begingroup$ I don't think that this would help much. What you would like assessing is how much the variation of that input would affect the output (by itself or in combination with other features) $\endgroup$
    – elachell
    Nov 7, 2018 at 17:02
0
$\begingroup$

You can also compute permutation importance of the input variables: https://scikit-learn.org/stable/modules/permutation_importance.html

It is model-agnostic and is applicable to measure importance of input variables for “black-box” models like neural networks.

$\endgroup$
1
  • $\begingroup$ This answer could be made more useful by expanding upon the method and its appropriate use in addition to the link. Quoting or referencing the most important part of the link would be useful too. See here for guidance on answering effectively: stats.stackexchange.com/help/how-to-answer "...Always quote the most relevant part of an important link, in case the external resource is unreachable or goes permanently offline." $\endgroup$
    – dlid
    Apr 30, 2021 at 3:33
0
$\begingroup$

What you describe is IMHO a simple and effective way to determine what inputs your model is most sensitive to.

However, 'sensitive' is not necessarily the same as 'important'.

For example if your model is very prone to overfitting issues then such a metric could easily lead you in the wrong direction: Those 'highly sensitive inputs' could then actually just mean 'highly important for easy overfitting'.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.