Do word embeddings impact neural network performance?

Question

Consider a simple neural network with word embeddings as inputs. Suppose $x$ is a one-hot binary vector representing a word. We can compute the embedding with $e = Wx$. Then we compute the first hidden layer of our neural network using these word embeddings $h = \sigma(Me)$.

I understand that the intermediate representation, $e$, is very useful. But in the end $h = \sigma(MWx)$ and $MW$ is just another matrix. I understand that their are more weights and different gradients involved with two matrices, but does this actually effect the performance of the neural network?

Answer 1

Word embeddings mostly help because they can be pre-trained in an unsupervised way on large amounts of text. As a result, NN learns continuous representation of words in a space where words with similar meaning are close to each other. Since NN activation function is continuous (sigmoid or simililar) such representations are inherently easier for NN to work with, and they also improve generalization to unknown words. Also "word projection layer" can save computation time, when properly implemented.

If you only train NN on small supervised dataset and use random initializations, then additional layer for "embeddings" probably won't help much (Collobert el al paper actually demonstrates this)