Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm building a generative text model, and the output of one of the final layers is a word embedding (vector) of the generated word. I'm left with the task of converting this vector back to the actual word.
Is there a good algorithm for doing this inversion? I'm thinking of using a fully-connected/dense layer, but then it's decoupled from the original (forward) embedding layer. Ideally, I'd think it's better to make use of parameters of the embedding layer somehow for the inversion.
There is no one 'right' way to turn wordvectors back into words. The issue is that the words themselves form a discrete set of points in the embedding space, and so the output of a model is very unlikely to be exactly equal to the location of any word.
Typically if your model emits a vector $v$ then interpreting it as a word is done by finding a word $w$ with embedding $v_w$ such that $d(v, v_w)$ is small, i.e. $v$ is 'close' to the embedding of $w$. Choosing the distance function $d$ is up to you, although typically the cosine similarity is used. Depending on the application, you could also consider showing the top-$k$ similar words to your wordvector, which could offer a bit more diversity.
Are you familiar with autoencoders? They are defined in terms of two networks: encoder and decoder, that are usually symmetrical. The general assumption is that to decode the data from the latent representation, you probably need similar kind of architecture, as was needed for encoding it. While re-using the weights from encoder in many cases would be possible and may seem reasonable, defining septate decoder network is more popular solution, because it is much simpler.
The embeddings are floating-point numbers, to translate them to words you need a function that will map the numbers to words. There are many ways how this can be achieved, e.g. with recurrent neural networks that create words byte by byte, or $n$-gram by $n$-gram, or predicting the one-hot encodings for the words (usually the number of words is huge, so you need approximate solutions), you may take into consideration the words that appear before or after the predicted word and use etc. There is no single best approach, because this is problem specific.
I do like the idea of using a decoder network at the end of your architecture. This allows you to train on one-hot encoded words. Then your output will be a probability distribution over the words in your vocabulary. You can then draw randomly from the probability distribution. In some ways, what you get is more of "a smear" of possible words. More probable words would have a higher chance of being selected.
This of course gives you a probabilistic response words. A second run of the algorithm, given that you didn't fix the seed, will result in slightly different output.
If you look at seq2seq models and transformers, the way this is usually done is by outputting a one-hot encoded vector using a softmax layer. Basically, the output will be a vector whose length is the size of the vocabulary.
