Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?

Question

I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine):

Note that (as is usually the case in Machine Learning) we think of the training data as given and fixed, and of the weights as variables we have control over. Hence, even though we can easily use backpropagation to compute the gradient on the input examples $x_i$, in practice we usually only compute the gradient for the parameters (e.g. $W$, $b$) so that we can use it to perform a parameter update. However, as we will see later in the class the gradient on $x_i$ can still be useful sometimes, for example for purposes of visualization and interpreting what the Neural Network might be doing.

Here are my specific questions:

1. By backpropagation to compute the gradient on the input examples, does that mean computing the partial derivative of the loss $J$ with respect to input $x$ (i.e. computing ${\partial J} / {\partial x}$) rather than the usual partial derivative with respect to the weights (i.e. computing $\partial{J} / \partial{W}$)?

2. In natural language processing, there is the concept of pre-trained word embeddings, which are vector representations of words. One can also fine-tune the word embeddings by further training on a specific task and letting the embeddings be updated. Is this embedding fine-tuning an example of backpropagation to compute the gradient on the input examples?

Thank you for any information.

Answer 1

If you view the back-propagation purely as a graph algorithm operating on the computation graph, the inputs are qualitatively the same nodes as the parameters - some floats that one can change and the output of the network will change. The back-propagation algorithm can compute the gradient of anything in the computation graph (given all operations are differentiable) regardless if it makes sense or not.

Usually, it does not make sense to change the inputs because you want a model that works well for inputs as they are. However, input gradients can be used for generating adversarial examples - small perturbations of the input that are hardly noticeable for a human but change the output of the model (more a topic in computer vision and speech recognition than NLP).

I think it can make sense to view embeddings fine-tuning as back-propagation to the input, but I think NLP people usually consider the embeddings to be more of model parameters than the inputs. The "real" inputs are one-hot vectors/vocabulary indices.

Answer 2

"it does not make sense to change the inputs because you want a model that works well for inputs as they are."

Well, if inputs are controllable parameters of the system then it makes a perfect sense to modify them... for example to control servomechanism to balance an inverted pendulum...