# Why do we not use continuously defined losses in NLP?

I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors over a vocabulary, which do not by themselves yield gradients, or, phrased differently, have no information about the similarity between words in the vocabulary.
Since we have continous representations of words via word embeddings, why do we not define a loss function on these?
For example, we could let our output layer produce the real valued embeddings of the target space, define the loss to be for example the negative dot product (or some other, perhaps more sophisticated metric) with the target embedding. To produce tokens in the vocabulary, we only need to choose the nearest neighbor during inference, for example. I might try this myself but the idea feels so basic that I wonder if somebody has done this?

• This is an interesting question (+1), but I do have one quibble. There definitely are gradients in these models even though the targets are 1-hot. The simplest example is a model is to maximize the likelihood of $p$ in a Bernoulli $\text{Bern}(p)$ model. Logistic regression, a specific case of a neural network, extends this concept to linear predictors of $p$.
– Sycorax
Commented Nov 9, 2020 at 15:33
• Right, I thought about this the wrong way I think. Of course a language model does have gradients from its predictions, when the gradient is taken with respect to NLL, for example. So the continuous target $\hat{y}$ being predicted is a conditional probability. Is there a reason the probability can not be substituted by some notion of distance in the target space? (I guess this is what Transformers do: Output layer weights are fixed to target embedding weights => the second to last layer essentially outputs target embeddings, no?) Commented Nov 9, 2020 at 23:28
• Some networks have losses defined in terms of distance inferred from the target. Triplet networks spring to mind as an example, but that's probably because I've used them a lot. Triplets are a cool idea, and I think an under-utilized one, but they can be tricky. Here's some discussion about why: stats.stackexchange.com/questions/475655/…
– Sycorax
Commented Nov 10, 2020 at 0:04