Consider a simple neural network with word embeddings as inputs. Suppose $x$ is a one-hot binary vector representing a word. We can compute the embedding with $e = Wx$. Then we compute the first hidden layer of our neural network using these word embeddings $h = \sigma(Me)$.

I understand that the intermediate representation, $e$, is very useful. But in the end $h = \sigma(MWx)$ and $MW$ is just another matrix. I understand that their are more weights and different gradients involved with two matrices, but does this actually effect the performance of the neural network?

  • $\begingroup$ I'm slowly warming up to this idea... this paper is helping me understand. I suppose you can save on the number of weights used in the model... $\endgroup$
    – Chet
    Feb 13, 2015 at 1:52

1 Answer 1


Word embeddings mostly help because they can be pre-trained in an unsupervised way on large amounts of text. As a result, NN learns continuous representation of words in a space where words with similar meaning are close to each other. Since NN activation function is continuous (sigmoid or simililar) such representations are inherently easier for NN to work with, and they also improve generalization to unknown words. Also "word projection layer" can save computation time, when properly implemented.

If you only train NN on small supervised dataset and use random initializations, then additional layer for "embeddings" probably won't help much (Collobert el al paper actually demonstrates this)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.