In word-embeddings visualization I often see that people perform the following two steps:
- Train high-dimensional embeddings on a corpus
- Use dimensionality reduction technique (e.g. PCA)
So why not simply set size of the hidden layer to 2, i.e. train word embeddings of size 2 in the first place, hence, eliminating the need for extra steps of dimensionality reduction like PCA. Is the information loss greater than one would experience by, say, using PCA reduction?