Consider a simple neural network with word embeddings as inputs. Suppose $x$ is a one-hot binary vector representing a word. We can compute the embedding with $e = Wx$. Then we compute the first hidden layer of our neural network using these word embeddings $h = \sigma(Me)$.
I understand that the intermediate representation, $e$, is very useful. But in the end $h = \sigma(MWx)$ and $MW$ is just another matrix. I understand that their are more weights and different gradients involved with two matrices, but does this actually effect the performance of the neural network?