Why do we not use continuously defined losses in NLP? I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors over a vocabulary, which do not by themselves yield gradients, or, phrased differently, have no information about the similarity between words in the vocabulary.
Since we have continous representations of words via word embeddings, why do we not define a loss function on these?
For example, we could let our output layer produce the real valued embeddings of the target space, define the loss to be for example the negative dot product (or some other, perhaps more sophisticated metric) with the target embedding.
To produce tokens in the vocabulary, we only need to choose the nearest neighbor during inference, for example.
I might try this myself but the idea feels so basic that I wonder if somebody has done this?
 A: You are indeed not the first one who thinks like this. The pragmatic answer would be: people tried, but the negative-log likelihood seems to work better.
There are several relatively successful attempts (mostly in machine translation):

*

*Von mises-fisher loss for training sequence to sequence models with continuous outputs


*A Margin-based Loss with Synthetic Negative Samples for Continuous-output Machine Translation


*Efficient Contextual Representation Learning Without Softmax Layer
I think the main drawback of the methods is that you need to have the word/symbol embeddings in advance, however, for high-resource tasks (such as MT or large-scale representation pretraining), it is usually better when you train the embeddings end-to-end with the rest of the model. Another issue is that SoTA models do not segment the text into standard word tokens, but use sub-word segmentation (BPE or SentencePiece) and current methods for word embeddings (such as Word2Vec or FastText) struggle to learn good embeddings for subwords.
So, I believe that if someone solves these two issues, what you propose will be the way to go.
