Following a question already answered (Extracting weight importance from One-Layer feed-forward network) I am looking for inference about relevance of inputs in neural networks.

Considering a deep net, where reconstructing the input importance by going backward through the layers from the output node of interest may be difficult or time consuming, I was wondering whether there was some theoretical framework in performing sensitivity analysis for neural network, basically slightly change an input and consider how the ouptut node of interest changes.

Is there a canonical way of performing some sort of sensitivity analysis in neural networks?

I would really welcome some Python code to do so, if there is any

  • $\begingroup$ import torch import torch.nn class DeepNet(torch.nn.Module): def __init__(self): super(DeepNet,self).__init__() self.layer = torch.nn.Sequential( torch.nn.Linear(3,8), torch.nn.ReLU(), torch.nn.Linear(8,1), torch.nn.Sigmoid() ) def forward(self, x): y = self.layer(x) return y net = DeepNet() test_x = torch.rand((5,3),requires_grad=True) test_y = torch.sin(test_x) loss_fn = torch.nn.MSELoss() pred_y = net(test_x) loss = loss_fn(pred_y,test_y) loss.backward() print("gradient of input variable:",test_x.grad.data) print("gradient of neurons in the first layer:",net.layer[0].weight.grad.data) print $\endgroup$
    – Kibak
    Commented Apr 13, 2020 at 1:29

2 Answers 2


The sensitivity analysis you suggest corresponds to examining the partial derivatives of the outputs with respect to the inputs. Say the output vector $y \in \mathbb{R}^m$ is given by $y= f(x)$ , where $x \in \mathbb{R}^d$ is the input vector and $f$ is the function the network implements. The Jacobian of the outputs w.r.t. the inputs is:

$$J_{ij}(x) = \frac{\partial}{\partial x_j} f_i(x)$$

The Jacobian gives the local rate of change of each output w.r.t. each input, so it tells us how $f$ will behave in response to infinitesimal perturbations. If we start with input $x$ and add an infinitesimal value $\Delta$ to the $j$th input, we expect the $i$th output to increase by $\Delta J_{ij}(x)$.

If $J_{ij}(x)$ has large magnitude, it means that output $i$ is sensitive to input $j$ in the vicinity of $x$. Because $f$ is, in general, nonlinear, this notion of sensitivity depends on the input; it may be large in some regions and near zero in others. If you want some kind of summary measure of how strongly the outputs depend on the inputs, you'd have to aggregate over multiple input values. For example, you could take the absolute value of the Jacobian, averaged over all inputs in the training set (which acts as a surrogate for the expected value w.r.t. the underlying distribution of inputs). Of course, this kind of summary will end up discarding information, so could be misleading in some circumstances.

You can use the chain rule to derive an expression for the Jacobian, similarly to how you'd derive the gradient of the loss function w.r.t. the parameters for use with backprop. You can also compute it using automatic differentiation, using a library like Theano, TensorFlow, etc. There's not much reason to perform finite differencing (i.e. actually simulate the perturbation and measure the change in output), unless the function your network implements is nondifferentiable (in which case the Jacobian doesn't exist).

A couple caveats: If the inputs have different units/scales than each other, the sensitivities will also have different units/scales, and can't be directly compared. Standardizing/scaling the inputs is one possible solution. It's also important to keep in mind is that this type of analysis tells us about the model itself, but not necessarily the underlying distribution that generated the data. For example, if two inputs are correlated, the model might end up using the first but not the second. In this case, we'd find that the sensitivity is high for the first input and low for the second, but should not conclude that the first input is inherently more important for predicting the output in general.

This article should be of interest.

  • $\begingroup$ great answer and great article! if anyone is interested in implementing this method, you can find a nice implementation of the jacobian calculation here: medium.com/unit8-machine-learning-publication/… $\endgroup$
    – pcko1
    Commented Mar 8, 2019 at 14:58

Although from a different community, the sensitivity of output of DNN to input is also of high interest of earth science or other applied physics field. Because it's more-or-less a representation of the interpretation of the neural network. It told us how the output can be influenced by the input. Please find my publication about it in data assimilation over here: https://www.sciencedirect.com/science/article/pii/S0034425722002309. We tries to make the DNN "physically plausible" by calculating the sensitivity (or Jacobian), and tried to see whether the sensitivity makes physical sense. We also find some problems about that and are addressing them. Just follow our work.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.