3
$\begingroup$

In word-embeddings visualization I often see that people perform the following two steps:

  1. Train high-dimensional embeddings on a corpus
  2. Use dimensionality reduction technique (e.g. PCA)

So why not simply set size of the hidden layer to 2, i.e. train word embeddings of size 2 in the first place, hence, eliminating the need for extra steps of dimensionality reduction like PCA. Is the information loss greater than one would experience by, say, using PCA reduction?

$\endgroup$
1
  • $\begingroup$ This is good question so basically you are saying why do we first train in high-dimension and then decrease dimension by using PCA. $\endgroup$
    – user0193
    Commented Mar 31, 2021 at 22:07

1 Answer 1

2
$\begingroup$

The use of a word embedding is almost never to be visualized. The visualization is just a diagnostic and interpretation tool. And 2 dimensions is just not enough in language. The only reason you want 2 dimensions is because that's what you can print on paper.

If you learn extremely low-dimensional embeddings, they're probably going to be useless. Consider the consequences of dimension-2 embeddings. There are only two axes on which meaning can vary, so you'll wind up with a lot of word pairs with similar dot products. It will be hard to properly distinguish between meanings.

My point is that your embeddings will be bad for whatever your task is (language modeling, translation, sentiment analysis, whatever) if there are so few dimensions to your embedding that it cannot capture major distinctions in language. That's the motivation for learning high-dimensional embeddings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.