In word-embeddings visualization I often see that people perform the following two steps:
So why not simply set size of the hidden layer to 2, i.e. train word embeddings of size 2 in the first place, hence, eliminating the need for extra steps of dimensionality reduction like PCA. Is the information loss greater than one would experience by, say, using PCA reduction?
The use of a word embedding is almost never to be visualized. The visualization is just a diagnostic and interpretation tool. And 2 dimensions is just not enough in language. The only reason you want 2 dimensions is because that's what you can print on paper.
If you learn extremely low-dimensional embeddings, they're probably going to be useless. Consider the consequences of dimension-2 embeddings. There are only two axes on which meaning can vary, so you'll wind up with a lot of word pairs with similar dot products. It will be hard to properly distinguish between meanings.
My point is that your embeddings will be bad for whatever your task is (language modeling, translation, sentiment analysis, whatever) if there are so few dimensions to your embedding that it cannot capture major distinctions in language. That's the motivation for learning high-dimensional embeddings.
