8
$\begingroup$

I'm building a generative text model, and the output of one of the final layers is a word embedding (vector) of the generated word. I'm left with the task of converting this vector back to the actual word.

Is there a good algorithm for doing this inversion? I'm thinking of using a fully-connected/dense layer, but then it's decoupled from the original (forward) embedding layer. Ideally, I'd think it's better to make use of parameters of the embedding layer somehow for the inversion.

$\endgroup$
8
$\begingroup$

There is no one 'right' way to turn wordvectors back into words. The issue is that the words themselves form a discrete set of points in the embedding space, and so the output of a model is very unlikely to be exactly equal to the location of any word.

Typically if your model emits a vector $v$ then interpreting it as a word is done by finding a word $w$ with embedding $v_w$ such that $d(v, v_w)$ is small, i.e. $v$ is 'close' to the embedding of $w$. Choosing the distance function $d$ is up to you, although typically the cosine similarity is used. Depending on the application, you could also consider showing the top-$k$ similar words to your wordvector, which could offer a bit more diversity.

$\endgroup$
1
  • $\begingroup$ That's a good point - the mapping to embeddings is not surjective. The algorithm you suggest definitely works, but because it's not "analytic" - requiring a search over the word space, it is a computationally expensive one. I can't think of a better approach though. $\endgroup$
    – rishai
    Aug 22 '19 at 19:10
3
$\begingroup$

Are you familiar with autoencoders? They are defined in terms of two networks: encoder and decoder, that are usually symmetrical. The general assumption is that to decode the data from the latent representation, you probably need similar kind of architecture, as was needed for encoding it. While re-using the weights from encoder in many cases would be possible and may seem reasonable, defining septate decoder network is more popular solution, because it is much simpler.

The embeddings are floating-point numbers, to translate them to words you need a function that will map the numbers to words. There are many ways how this can be achieved, e.g. with recurrent neural networks that create words byte by byte, or $n$-gram by $n$-gram, or predicting the one-hot encodings for the words (usually the number of words is huge, so you need approximate solutions), you may take into consideration the words that appear before or after the predicted word and use etc. There is no single best approach, because this is problem specific.

$\endgroup$
4
  • $\begingroup$ I think to implement a decoder (to word) of an autoencoder, you are still faced with the challenge of inverting the Embedding model. Because of this, from the few implementations I've looked at, autoencoders sidestep the problem somehow - by working at the character level, or by using one-hot encoding, or by training to reproduce the embeddings (as decoding to word is often unnecessary). $\endgroup$
    – rishai
    Aug 22 '19 at 19:08
  • $\begingroup$ @rishai this is exactly what you would do: either use RNN on bytes, characters, n-grams etc., or predict one-hot encoded words. What else would you imagine? $\endgroup$
    – Tim
    Aug 22 '19 at 19:11
  • $\begingroup$ I had no other ideas, frankly. But I wondered if there is a better, or canonical solution... $\endgroup$
    – rishai
    Aug 22 '19 at 23:28
  • $\begingroup$ Just happened to notice your edits; I don't get notified of edits, for some reason. I'll take a look at that link, thanks. $\endgroup$
    – rishai
    Aug 26 '19 at 0:47
1
$\begingroup$

I do like the idea of using a decoder network at the end of your architecture. This allows you to train on one-hot encoded words. Then your output will be a probability distribution over the words in your vocabulary. You can then draw randomly from the probability distribution. In some ways, what you get is more of "a smear" of possible words. More probable words would have a higher chance of being selected.

This of course gives you a probabilistic response words. A second run of the algorithm, given that you didn't fix the seed, will result in slightly different output.

$\endgroup$
1
$\begingroup$

If you look at seq2seq models and transformers, the way this is usually done is by outputting a one-hot encoded vector using a softmax layer. Basically, the output will be a vector whose length is the size of the vocabulary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.