in machine learning, using One-Hot encoded data like words for RNNs (especially when we have a large dictionary like 10M or larger?) causes the dot product with the weight matrix slow.
THE QUESTION IS: Is this true? and if yes, Is one-hot encoding still used even though?
and, Does word embedding algorithms use one-hot encoding to represent data in training?
No one actually computes the dot product explicitly, since with a one-hot encoding you just need to read off the corresponding row/column of the matrix you're multiplying with (which takes constant time and is very fast). Otherwise it's prohibitively expensive and extremely slow.
If your vocabulary size is very large (i.e. millions), then usually people use word-embeddings like word2vec, Glove, etc as inputs for RNNs and other models. However, to train these embeddings, you do one-hot encoding for your words. In word2vec, you randomly sample word/context pairs, and you never actually multiply the one-hot encoding as explained above. You hash your word vectors, so after selecting a pair of words, you look up their vectors, and then do a gradient descent step. So the only dot products you end up computing are with respect to the in/out embeddings of your words.
