Do word embeddings impact neural network performance?

Consider a simple neural network with word embeddings as inputs. Suppose $$x$$ is a one-hot binary vector representing a word. We can compute the embedding with $$e = Wx$$. Then we compute the first hidden layer of our neural network using these word embeddings $$h = \sigma(Me)$$.

I understand that the intermediate representation, $$e$$, is very useful. But in the end $$h = \sigma(MWx)$$ and $$MW$$ is just another matrix. I understand that their are more weights and different gradients involved with two matrices, but does this actually effect the performance of the neural network?

• I'm slowly warming up to this idea... this paper is helping me understand. I suppose you can save on the number of weights used in the model... – Chet Feb 13 '15 at 1:52