I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors over a vocabulary, which do not by themselves yield gradients, or, phrased differently, have no information about the similarity between words in the vocabulary.
Since we have continous representations of words via word embeddings, why do we not define a loss function on these?
For example, we could let our output layer produce the real valued embeddings of the target space, define the loss to be for example the negative dot product (or some other, perhaps more sophisticated metric) with the target embedding.
To produce tokens in the vocabulary, we only need to choose the nearest neighbor during inference, for example.
I might try this myself but the idea feels so basic that I wonder if somebody has done this?
-
$\begingroup$ This is an interesting question (+1), but I do have one quibble. There definitely are gradients in these models even though the targets are 1-hot. The simplest example is a model is to maximize the likelihood of $p$ in a Bernoulli $\text{Bern}(p)$ model. Logistic regression, a specific case of a neural network, extends this concept to linear predictors of $p$. $\endgroup$– Sycorax ♦Commented Nov 9, 2020 at 15:33
-
$\begingroup$ Right, I thought about this the wrong way I think. Of course a language model does have gradients from its predictions, when the gradient is taken with respect to NLL, for example. So the continuous target $\hat{y}$ being predicted is a conditional probability. Is there a reason the probability can not be substituted by some notion of distance in the target space? (I guess this is what Transformers do: Output layer weights are fixed to target embedding weights => the second to last layer essentially outputs target embeddings, no?) $\endgroup$– infoboxCommented Nov 9, 2020 at 23:28
-
1$\begingroup$ Some networks have losses defined in terms of distance inferred from the target. Triplet networks spring to mind as an example, but that's probably because I've used them a lot. Triplets are a cool idea, and I think an under-utilized one, but they can be tricky. Here's some discussion about why: stats.stackexchange.com/questions/475655/… $\endgroup$– Sycorax ♦Commented Nov 10, 2020 at 0:04
1 Answer
You are indeed not the first one who thinks like this. The pragmatic answer would be: people tried, but the negative-log likelihood seems to work better.
There are several relatively successful attempts (mostly in machine translation):
Von mises-fisher loss for training sequence to sequence models with continuous outputs
A Margin-based Loss with Synthetic Negative Samples for Continuous-output Machine Translation
Efficient Contextual Representation Learning Without Softmax Layer
I think the main drawback of the methods is that you need to have the word/symbol embeddings in advance, however, for high-resource tasks (such as MT or large-scale representation pretraining), it is usually better when you train the embeddings end-to-end with the rest of the model. Another issue is that SoTA models do not segment the text into standard word tokens, but use sub-word segmentation (BPE or SentencePiece) and current methods for word embeddings (such as Word2Vec or FastText) struggle to learn good embeddings for subwords.
So, I believe that if someone solves these two issues, what you propose will be the way to go.
-
$\begingroup$ Awesome, the first paper is pretty much what I was thinking of. As with all new approaches, I think there is also huge room for improvement, especially different losses such as in the second paper. The problem with needing to use pre trained target embeddings I still dont understand after reading the first paper, though. Why cant I update the target embedding matrix using chain rule like always? $\endgroup$– infoboxCommented Nov 10, 2020 at 16:38