All Questions
7 questions
1
vote
1
answer
388
views
Why the Transformer model does not require negative sampling but word2vec does?
Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side.
For word2vec models, negative sampling is used for computational reasons:
Is negative sampling ...
- 2,485
1
vote
0
answers
57
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 11
0
votes
1
answer
546
views
Word2vec/SkipGram: Why softmax?
In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling.
But if in the ...
- 103
2
votes
1
answer
433
views
How does Word2Vec CBOW softmax work with multiple context words?
I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4):
$$
p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
- 123
3
votes
1
answer
1k
views
Does hierarchical softmax of skip gram and CBOW only update output vectors on the path from the root to the actual output word?
After reading word2vec Parameter Learning Explained by Xin Rong, I understand that in the hierarchical softmax model, there is no output vector representation for words, instead, each of the $V-1$ ...
- 620
3
votes
1
answer
1k
views
Why would the sampled softmax work? [word2vec]
I'm following Udacity Deep learning course and the instructor briefly explained about the "Sampled Softmax" used in word2vec.
In tensorflow word2vec_basic.py, the implementation is like below.
<...
- 879
16
votes
3
answers
11k
views
Why is hierarchical softmax better for infrequent words, while negative sampling is better for frequent words?
I wonder why hierarchical softmax is better for infrequent words, while negative sampling is better for frequent words, in word2vec's CBOW and skip-gram models. I have read the claim on https://code....
- 47.6k