Most of the explainability methods use Gradient for measuring the sensitivity of the model to the input features, why can't we use the activation functions themselves as a measure of sensitivity? After all, gradient is basically derivation of the original function.
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
The idea with gradient-based interpretability techniques is that you want to measure the contribution of each input feature towards the final classification. The advantage of these techniques are that they do this in one pass.
Let's say for example you have an input image $x$ that consists $32 \times 32$ pixels. Your goal here is to see how much each input pixel affects the output prediction (i.e. $\hat y$) of the model. The contribution of the input pixel $x_{i,j}$ to the final prediction can be found by computing the partial derivative of the output w.r.t this pixel.
$$ \frac{\partial \hat y}{\partial x_{i, j}} $$
The gradient of the is the vector of the contributions of all the input pixels.
$$ \Delta_x\hat y = \frac{\partial \hat y}{\partial x}, \\ where \;\;x=\{x_{i, j}\}, \\ i \in \{1, ..., 32\}, \;\;j \in \{1, ..., 32\} $$
The activations by themselves (without the gradient) give no information regarding which input caused them to activate more!
Note: there are also other techniques for model interpretability (e.g. perturbing the input and seeing how this affects the output, class activation mapping), but I think your question refers only to gradient-based methods.
$\begingroup$
$\endgroup$
As you may know, the explainability methods, are a work in progress, and new methods appear regularly. As djib reminds us, gradient-based explanations provide an interpretation of the behaviour of the model in case of infinitesimal perturbation (not necessarily feasible). Among the other methods actually availables, there is indeed a lot of gradient based :
Vanilla Gradient (https://arxiv.org/abs/1312.6034)
Gradient $\circ$ Input
SmoothGrad (https://arxiv.org/abs/1706.03825)
Integrated Gradient (https://arxiv.org/abs/1703.01365)
For example, with Integrated gradient we perform an interpolation between a so-called baseline state (e.g. pixels at zero) and the current point. We evaluate the change of the score when adding each feature progressively (weighting by the distance of the feature from the baseline state). with $\tilde{x}$ the interpolated point, $\bar{x}$ the baseline state and $S_c(x)$ the score obtained for a class $c$. $$ IG := (x_i - \bar{x_i}) \circ \int_{\alpha=0}^1 { \frac{\partial S_c(\tilde{x})}{\partial \tilde{x_i} } \bigg\rvert_{ \tilde{x} = \bar{x} + \alpha(x - \bar{x}) }} $$
But there are other techniques that are not based only on the gradient, I advise you to look at the Grad-Cam method which is one of the most used technique today for explicability (since it is one of the methods that passes the sanity checks). This technique consist in $(1)$ using gradients to produce a weight for the feature map activations, and then $(2)$ we spatially sum the feature map activations after weighting them, so it combine both the gradient and the feature map activations.
More precisely, to obtain the localization map for a class $L_c$, we need to compute the weights $\alpha_k^c$ associated to each of the feature map activation $A^k$. As we use the last convolutionnal layer, $k$ will be the number of filters, $Z$ is the number of pixels in each feature map (width * height) .
$$ (1) \ \ \ \alpha_k^c = \frac{1}{Z} \sum_i\sum_j \frac{ \partial{y^c}} {\partial{A_{ij}^c} } $$ $$ (2) \ \ \ L^c = max(0, \sum_k \alpha_k^c A^k) $$
Notice that the size of the explanation depends on the size (height, width) of the last feature map, so we have to interpolate in order to find the same dimensions as the input.
Note: you could also find a guided-relu variant where to calculate the weighting, we use only the positive gradient whose feature map is also positive, this is how tf-explain library implement grad-cam.
$\begingroup$
$\endgroup$
Depending on what the purpose of the explanation is, gradients make sense (or do not make sense). To explain a decision, e.g., what parts of the inputs are relevant (as done in GradCAM) one needs the final decision and it makes a lot of sense to go backwards from the output (class).
For other purposes, gradients are not required, for example, if you wish to understand what information on the input is relevant for classification. To this end, one can pick a particular layer and apply a decoder on the activations that aims at reconstructing the original input. This allows to see what aspects of the inputs can be reconstructed well and which cannot. See ["Explaining Classifiers by Constructing Familiar Concepts", Schneider et al, Machine Learning, 2022] (Disclaimer, I am one of the authors)
