2
$\begingroup$

One-hot encoding as raw inputs for deep learning models can find its applications in many domains, such as bioinformatics, NLP, chemistry and so on. Suppose we trained a neural network $f(x)$ with $x$ one-hot encoded. Now I want to evaluate the importance of each character based on the gradient $\partial f(x)/\partial x$ (e.g. saliency, inputxgrad, integrated gradients ...). When training $f(x)$, the gradient of loss function is well-defined on the network weights. Then primary question here is if $f(x)$ is differentiable w.r.t. $x$?

Strictly speaking, $f(x)$ is defined on binary values. Then for instance, in the following figure, a small deviation in the position of "T" would make no sense. So $\partial f(x)/\partial x$ is not well-defined, is that correct?

one-hot encoded input

In the case of NLP, one-hot encoded words are first represented by embedding vectors of continuous values, e.g. word2vec. Then for a trained language model, for evaluating word contribution, we don't need to trace back to one-hot encoding but only to embedding vectors.

I haven't found similar discussions after a quick search. Is this trivial? Thanks a lot for your inputs!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In NLP, I have seen it done with one-hot encoding: https://colab.research.google.com/github/AndreasMadsen/python-textualheatmap/blob/master/notebooks/huggingface_bert_example.ipynb

But I've seen more places use embedding, then normalize the embedding to get a single score per token. This recent survey of input saliency shows better results for aggregating using the L2 norm: https://arxiv.org/pdf/2009.13295.pdf (as opposed to averaging). I also believe that (embedding) is what the Captum interpretability library uses. Examples:

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks a lot for the references. Technically there is no issue to actually compute $\partial y/\partial x$ in either PyTorch or Tensoflow framework. My concern is that what the calculated gradient values corresponding to 0's in the one hot encoding actually mean. In the first link, an euclidean norm is calculated for the gradient for each one hot encoded word. How to justify the effectiveness of this method? I would like to have conceptual justifications. $\endgroup$
    – doubllle
    Commented Oct 31, 2020 at 20:42
  • $\begingroup$ I might have found something. Please let me know if I understand this correctly: arxiv.org/pdf/1412.6815.pdf Top of page 5 (where it says "where Is ∈ R |V |×|s| is a matrix whose columns are 1-hot vectors identifying the dictionary index of each word in the sentence s"): This points that the one-hot vector is the same as the dot-product of Gradient * Embedding. Do you understand it like I do, or did I impose that understanding there myself, lol? $\endgroup$
    – Jay Alammar
    Commented Nov 2, 2020 at 12:28
  • $\begingroup$ I understood it the same way. Once we calculated the gradients wrt one-hot encoding, we can come up several ways of aggregating each word, a simple dot product is a straightforward way to do that, that means simply ignores values from 0's. But when you actually look at such gradients, you might doubt if this makes sense as large gradients can be observed from 0's. Following that, if we interpret that as "not having this word at this position is important", then this would complicate the whole analysis. $\endgroup$
    – doubllle
    Commented Nov 2, 2020 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.