Questions tagged [word2vec]
Word2vec is a neural network that represents words as vectors in a high dimensional space.
207 questions
63
votes
2
answers
40k
views
Should I normalize word2vec's word vectors before using them?
After training word vectors with word2vec, is it better to normalize them before using them for some downstream applications? I.e what are the pros/cons of normalizing them?
- 47.6k
61
votes
5
answers
38k
views
Apply word embeddings to entire document, to get a feature vector
How do I use a word embedding to map a document to a feature vector, suitable for use with supervised learning?
A word embedding maps each word $w$ to a vector $v \in \mathbb{R}^d$, where $d$ is some ...
- 6,738
43
votes
5
answers
25k
views
LDA vs word2vec
I am trying to understand what is similarity between Latent Dirichlet Allocation and word2vec for calculating word similarity.
As I understand, LDA maps words to a vector of probabilities of latent ...
- 5,896
38
votes
4
answers
53k
views
Interpreting negative cosine similarity
I was using the GLOVE model which is pre-trained by Stanford NLP group (link) and noticed that my similarity results showed some negative numbers. Upon inspecting the word-vector data file, I realized ...
- 616
28
votes
1
answer
19k
views
How does negative sampling work in word2vec?
I have been trying hard to understand the concept of negative sampling in the context of word2vec. I am unable to digest the idea of [negative] sampling. For example in Mikolov's papers the negative ...
- 383
25
votes
3
answers
13k
views
Why is skip-gram better for infrequent words than CBOW?
I wonder why skip-gram is better for infrequent words than CBOW in word2vec. I have read the claim on https://code.google.com/p/word2vec/.
- 47.6k
23
votes
2
answers
14k
views
How is the .similarity method in SpaCy computed?
Not Sure if this is the right stack site, but here goes.
How does the .similiarity method work?
Wow spaCy is great! Its tfidf model could be easier, but w2v with only one line of code?!
In his 10 ...
- 521
16
votes
3
answers
11k
views
Why is hierarchical softmax better for infrequent words, while negative sampling is better for frequent words?
I wonder why hierarchical softmax is better for infrequent words, while negative sampling is better for frequent words, in word2vec's CBOW and skip-gram models. I have read the claim on https://code....
- 47.6k
15
votes
3
answers
23k
views
What does average of word2vec vector mean?
The paper that I am reading says,
tweet is represented by the average of the word embedding vectors of
the words that compose the tweet.
Does this mean each word in the tweet (sentence) has to ...
- 151
14
votes
3
answers
13k
views
How does Word2Vec's skip-gram model generate the output vectors?
I am having problems understanding the skip-gram model of the Word2Vec algorithm.
In continuous bag-of-words is easy to see how the context words can "fit" in the Neural Network, since you basically ...
- 905
13
votes
2
answers
30k
views
How to determine parameters for t-SNE for reducing dimensions?
I am very new to word embeddings. I want to visualize how the documents are looking after learning. I read that t-SNE is the approach to do it. I have 100K documents with 250 dimensions as size of the ...
- 617
13
votes
2
answers
40k
views
How to use pre trained word2vec model?
Where can I find a reliable word2vec model trained on some English articles?
I need a word2vec black box, where I, for example, ...
- 627
12
votes
2
answers
21k
views
Handling unknown words in language modeling tasks using LSTM
For a natural language processing (NLP) task one often uses word2vec vectors as an embedding for the words. However, there may be many unknown words that are not captured by the word2vec vectors ...
- 5,146
12
votes
3
answers
17k
views
What is difference between keras embedding layer and word2vec?
In other words, is there a paper that describes the method of keras embedding layer? Is there a comparison between these methods (and other methods like Glove etc.)?
- 1,017
11
votes
1
answer
12k
views
Gradients for skipgram word2vec
I am going through the problems in the Stanford NLP deep learning class's written assignment problems http://cs224d.stanford.edu/assignment1/assignment1_soln
I am trying to understand the answer for ...
- 113
11
votes
1
answer
9k
views
Derivative of cross entropy loss in word2vec
I am trying to work my way through the first problem set of the cs224d online stanford class course material and I am having some issues with problem 3A: When using the skip gram word2vec model with ...
- 213
11
votes
1
answer
667
views
Word embedding algorithms in terms of performance
I'm trying to embed roughly 60 million phrases into a vector space, then calculate the cosine similarity between them. I've been using sklearn's CountVectorizer ...
- 213
10
votes
1
answer
5k
views
Input vector representation vs output vector representation in word2vec
In word2vec's CBOW and skip-gram models, how does choosing word vectors from $W$ (input word matrix) vs. choosing word vectors from $W'$ (output word matrix) impact the quality of the resulting word ...
- 47.6k
10
votes
2
answers
8k
views
What is the function that is being optimized in word2vec?
The following question is about Skipgram, but it would be a plus (though not essential) to answer the question for the CBOW model as well.
Word2Vec uses neural networks, and neural networks learn by ...
- 1,460
10
votes
2
answers
2k
views
What is the relation of the negative sampling (NS) objective function to the original objective function in word2vec?
I was reading the standard/famous word2vec model and according to standord's notes for cs224n the objective function changes from:
$$J_{original} = -\sum^{2m}_{j=0,j\neq m} u^\top_{c-m+j} v_c + 2m ...
- 7,074
9
votes
3
answers
11k
views
Can I apply word2vec to find document similarity?
I appreciate word2vec is used more to find the semantic similarities between words in a corpus, but here is my idea.
Train the word2vec model on a corpus
For each document in the corpus, find the ...
- 1,334
8
votes
2
answers
4k
views
Word2Vec : Interpretation of Subtraction or addition of vectors
I am curious, what does subtracting vectors, as in [man – woman] do in regards to Google's word2vec calculation of analogy ? Is this a measure of how different the two vectors are? So is
man – woman (...
- 8,850
8
votes
2
answers
1k
views
word2vec neural network - bias units
I am trying to get my head around word2vec (paper) and the underlying Skip-gram model. I hope I got the basics and intuition, but I am still not sure whether bias units are used in the input and/or in ...
- 391
7
votes
1
answer
3k
views
Word2Vec : Difference between the two Weight matrices
In Word2Vec algorithm, two weight matrices are learnt :
W : Input-hidden layer matrix
W': Hidden-output layer matrix
For reference, CBOW model architecture:
Why is W chosen to represent the word ...
- 73
7
votes
1
answer
4k
views
What is Contextual Embedding?
I understand word embeddings and word2vec.
In this paper: https://arxiv.org/pdf/1603.01547.pdf
they are saying a new type of word embedding.
...
- 439
7
votes
1
answer
2k
views
Do we use maximum likelihood or cross entropy Loss for training skip-gram model?
In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where
$$J(\theta) = -\frac{1}{T}\sum_{t=1}^...
- 1,116
7
votes
2
answers
2k
views
Word2vec that can distinguish words with different meanings
The word2vec is a very successful method for converting different words into a dense vector of real numbers. After learning, it comes up with a look-up table which you can use to obtain the vector ...
- 3,524
7
votes
3
answers
3k
views
Does it make sense to multiply two embedding vectors?
Many researchers are using neural network to infer embedding vectors for words, users, or items. Word embeddings, e.g., word2vec, allow people to calculate sum, average, and difference over embeddings....
- 2,095
7
votes
3
answers
1k
views
Do word vectors obtained via word embedding techniques really form a vector space?
Word embedding refers to feature learning techniques in natural
language processing where words are mapped to vectors of real numbers
in a low-dimensional space, the embedding space.
Similar to ...
- 1,631
7
votes
2
answers
8k
views
multi-class classification with word2vec
My problem: The input data is a corpus of short documents (a few sentences each). In each document some expressions need to be classified to categories. A document must contain some categories (each ...
- 219
7
votes
0
answers
730
views
Understanding Object2Vec
AWS released an interesting feature as part of the SageMaker service called Object2Vec that lets you make an embedding for search out of pretty much anything: documents, users, products, ...
- 6,767
6
votes
1
answer
2k
views
Understanding Word2Vec
I am trying to understand the word2vec algorithm (Mikolov et. al) but there are a few thing which I do not understand.
I get that the activation from the input layer ot the hidden layer is linear and ...
- 337
6
votes
2
answers
499
views
Confusion on how skip gram implementation is formulated
I'm using this source to understand the skip gram model. Let's say the context size is $4$ ($2$ context words on each side of the target word). This image illustrates how training examples are ...
- 750
6
votes
2
answers
1k
views
Why is the output weights matrix initialized in a different way as the word embedding matrix in word2vec?
I was plagued by this problem while reading the tensorflow tutorial. There the word embedding is intialized as follows:
...
- 6,912
6
votes
1
answer
1k
views
Exact details of how word2vec (Skip-gram and CBOW) generate input word pairs
I am trying to reimplement skip-gram and CBOW. I think I understand the architecture well, but I am confused on how to input pairs are exactly generated.
For skip-gram, based on McCormick's post, a ...
- 1,176
6
votes
2
answers
2k
views
Word2Vec models for irrelevant word order
I'm searching for a ready-to-use model, preferably in TensorFlow, that learns embeddings for words from a corpus, but without taking word order into account.
So far I have a
vocabulary of 8822 words
...
- 71
6
votes
0
answers
2k
views
Why word embeddings learned from word2vec are linearly correlated
I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino ...
- 1,176
5
votes
2
answers
10k
views
merging two word embedding models?
I have a task for which I am using word embeddings but because limited data, I can't train.
For one part of my task Word2Vec model of Google News is working okay while for the other one, Glove ...
- 617
5
votes
1
answer
6k
views
How to train sentence/paragraph/document embeddings?
I'm well aware of word embeddings (word2vec or Glove) and I know of four papers treating the subject of more general embeddings :
Distributed Representations of Sentences and Documents - Quoc V. Le,...
- 120
5
votes
1
answer
6k
views
What does word embedding weighted by tf-idf mean?
The paper that I am reading explains about how it implemented the feature vector used for a twitter sentiment classification task.
The first is a simple combination, where each tweet is represented ...
- 51
5
votes
3
answers
1k
views
In word2vec, for analogies do we use "in" or "out" vectors?
In word2vec each word is associated with two vectors (one for in and one for out) so that it predicts conditional probability:
$$P(word_{out}|word_{in}) = \frac{\exp(v_{in} \cdot \tilde{v}_{out})}{\...
- 5,896
5
votes
1
answer
1k
views
glove: king - man + woman != queen
I downloaded 'glove.twitter.27B.25d.txt' from here https://nlp.stanford.edu/projects/glove/, and out of curiosity I wanted to see if king - man + woman does indeed ...
- 2,721
5
votes
1
answer
754
views
Why have a tanh layer, max pooling layer and then another tanh layer
I have been reading a Facebook paper, read here, and am confused about certain features of the architecture. I do not understand why they have a tanh layer, max-pooling layer, and then another tanh ...
- 51
4
votes
1
answer
932
views
Word2Vec vs. Doc2Vec Word Vectors
I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access.
My ...
- 1,602
4
votes
1
answer
462
views
Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?
I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
- 1,027
4
votes
1
answer
6k
views
How to use multiple words token with gensim word2vec?
I'm using pre-trained word2vec model lexvec.enwiki+newscrawl.300d.W.pos.vectors with gensim
this model "knows" a lot of words, but it doesn't know things like this: "great britain" or "star fruit"
...
- 627
4
votes
2
answers
3k
views
What exactly are "input" and "output" word representations?
So, I was reading Distributed Representations of Words and Phrases and their Compositionality, and I can't understand this part on page 3:
What exactly are these representations? One-hot vectors? ...
user150214
4
votes
1
answer
811
views
Why is the softmax on the dot product of the word embedding is the probability of context given word?
I was learning about the Word2Vec model, and the following equation was shown:
$\huge{p(o|c) = \frac{exp(u^T_ov_c)}{\sum_{w\in{V}}exp(u^T_wv_c)}}$
in words, the probability of the context word given ...
- 273
4
votes
1
answer
4k
views
Is skip-gram model in word2vec an expanded version of N-Gram model? skip-gram vs. skip-grams?
The skip-gram model of word2vec uses a shallow neural network to learn the word embedding with (input-word, context-word) data. When I read the tutorials for the skip-gram model there was not any ...
- 2,485
4
votes
1
answer
2k
views
Word2Vec - Why do we take input-hidden layer weights as word embeddings
I am currently trying to understand how the Word2Vec neural network works, but do not understand why we choose to take the weight vectors between the input and hidden layer as our word embedding ...
- 291