Questions tagged [word2vec]
Word2vec is a neural network that represents words as vectors in a high dimensional space.
207 questions
2
votes
0
answers
35
views
Example orthonormal basis of Word Embedding Space?
Models such as Word2Vec supposedly provide a bijection between language tokens and some "latent-space" that is in fact a high-dimensional vector space.
If this is a vector space, it should ...
- 21
0
votes
0
answers
40
views
How does the chain-rule look for the gradient of a loss function?
When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like:
$...
- 1
1
vote
0
answers
20
views
Team vectors similar to word vectors for a corpus of text
Is there any way I could iteratively create a set of vectors, similar to vectors when embedding words through word2vec, that could represent vectors between teams, and also capture information about a ...
- 11
2
votes
0
answers
50
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 66
1
vote
0
answers
173
views
When training word2vec, why is a new negative sampling process formulated instead of simply downsampling?
(For background, see The Skip-Gram Model.1 This question does not exactly use their notation, but you should be able to follow along.)
The original skip-gram log-likelihood of a single word, $w$, ...
- 1,625
0
votes
1
answer
44
views
Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?
In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
- 31
1
vote
1
answer
120
views
How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?
I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
- 31
1
vote
1
answer
226
views
How does training word embeddings bring similar words closer together?
How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
- 485
1
vote
1
answer
388
views
Why the Transformer model does not require negative sampling but word2vec does?
Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side.
For word2vec models, negative sampling is used for computational reasons:
Is negative sampling ...
- 2,485
1
vote
0
answers
57
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 11
0
votes
1
answer
336
views
Why should the weight matrix encode word embeddings in CBOW/skip-gram?
Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings.
I'm looking for an intuitive ...
1
vote
0
answers
229
views
Skip gram model and negative log loss likelihood
I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
- 23
2
votes
1
answer
58
views
Text similarity for badly written text
Consider the following scenario:
Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
- 337
2
votes
1
answer
649
views
Should I use cosine or dot similarity inside word2vec's neural network?
I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
- 185
1
vote
1
answer
340
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
- 41
0
votes
0
answers
47
views
Continuous Bag of Words NY Time Corpus
I am working to implement the continuous bag of words approach on the New York Times corpus dataset. However, I am getting word embeddings that do not seem very useful based on a few examples of ...
- 51
0
votes
1
answer
193
views
What is a better way to classify text using word2vec?
I am using word2vec to classify documents into various categories. Let's say we have a document:
Thousands of people with student loan debt will have their debt ...
- 101
1
vote
0
answers
150
views
Using doc2vec embeddings as model input our perhaps similarity comparison? [closed]
Doc2vec is an extension of word2vec, which creates vector representations of documents.
One can use this representations as input to some classifier/regression(Logistic Regression, XGboost, LightGBM .....
- 251
1
vote
2
answers
718
views
What makes training time longer with bigger parameter size in a deep learning model?
I try to understand, is it always the case with the more parameter you trained, the more training time you need when training a deep learning model. For example, i have a CNN model for text ...
- 121
1
vote
1
answer
105
views
Ensure trained word embeddings get high similarity with particular words
I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times
...
- 1,097
1
vote
1
answer
621
views
Does not 'removing the stopwords' affect Natural Language Processing Results in a high degree?
Most stopword lists contain contradicting prepositions (before-after, into-out of) and negativity words (not, no). Removing such words from the text almost always changes the meaning drastically.
The ...
- 129
0
votes
1
answer
546
views
Word2vec/SkipGram: Why softmax?
In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling.
But if in the ...
- 103
0
votes
0
answers
81
views
How the Negative Sampling algorithm chooses the negative samples (k) in character-based embeddings for the Word2Vec model?
In the context of word-based embeddings, the Negative Sampling algorithm chooses negative samples (k) from the most frequent words in the corpora which usually present less meaningful information than ...
MManahi
1
vote
0
answers
536
views
Skip-gram gradient with respect to outer word vector
For the skip gram model we know the naive softmax loss is as follows;-
$$ J=-\log(p(o \mid c)) = -\log\left(\frac {\exp\left(u_{o}^{T} v_{c}\right)}{\sum_{w=1}^{W}\exp\left(u_{w}^{T} v_{c}\right)}\...
- 131
0
votes
0
answers
29
views
Word-embedding does not provide expected relations between words
I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
2
votes
1
answer
433
views
How does Word2Vec CBOW softmax work with multiple context words?
I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4):
$$
p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
- 123
1
vote
0
answers
306
views
Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?
I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer).
However, ...
- 7,074
7
votes
1
answer
2k
views
Do we use maximum likelihood or cross entropy Loss for training skip-gram model?
In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where
$$J(\theta) = -\frac{1}{T}\sum_{t=1}^...
- 1,116
1
vote
1
answer
279
views
Extracting word embedding features of a sentence using Transformer-XL
As you know, there are several pre-trained models that we can use to extract word embeddings.
As an example, I can use the following codes to retrieve word2vec features of my text:
...
- 395
2
votes
1
answer
2k
views
Distance measure in word2vec
I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation.
I have often read that the ...
- 131
1
vote
1
answer
330
views
Intuition for GloVe word embeddings
I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used.
The ...
- 1,116
4
votes
1
answer
810
views
Why is the softmax on the dot product of the word embedding is the probability of context given word?
I was learning about the Word2Vec model, and the following equation was shown:
$\huge{p(o|c) = \frac{exp(u^T_ov_c)}{\sum_{w\in{V}}exp(u^T_wv_c)}}$
in words, the probability of the context word given ...
- 273
4
votes
1
answer
931
views
Word2Vec vs. Doc2Vec Word Vectors
I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access.
My ...
- 1,602
1
vote
0
answers
99
views
Can I skip the Keras Embedding Layer if I already transformed the data to Word2Vec (Google News 300 format)?
Trying to do sentiment analysis with an LSTM NN. I think I understand what the embedding layer does: map each word to a fixed-di-vector.
However, previously, for each text sample, I transformed each ...
- 11
0
votes
1
answer
286
views
When to use documents vs. sentences for Word2Vec?
I have a collection of words from different communities. Each community has a different way of using language and will provide a different word embedding. I can concatenate the sentences from the ...
- 3
2
votes
1
answer
95
views
Debiasing word embeddings
I'm reading the paper titled "man to computer programmer is woman to homemaker. Debiasing word embeddings". I'm right now trying to figure out the math and logic behind it and was doing OK ...
- 153
4
votes
1
answer
462
views
Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?
I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
- 1,027
1
vote
0
answers
117
views
NLP for customer reviews and summaries
I'm trying to develop a model in R that will compare a customer review with a summary of that review that is completed by an employee. The purpose is to ensure that the employee is accurately tagging ...
- 11
3
votes
1
answer
1k
views
Word embeddings - Pre-trained tokenizers vs more involved methods
I'm drowning under all the various methods of converting my text corpora into embeddings.
I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using ...
- 205
2
votes
0
answers
639
views
What is GloVe model's loss function actually minimizing?
I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
- 1,630
2
votes
3
answers
437
views
How can "word hashing" cause a collision in DSSM?
They say in their paper, that "word hashing" can cause a collision. But I don't understand, how. For example, if word good is tranformed to ...
- 412
1
vote
1
answer
1k
views
Can Latent Dirichlet Allocation (LDA) be used to generate word embeddings?
In the original Word2Vec paper (Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013), I came across this phrase:
Many different types of models were proposed for ...
- 113
2
votes
1
answer
1k
views
Generating Sentence Vectors from Word2Vec
I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec.
I read lot of materials and found that Averaging the ...
- 131
1
vote
1
answer
177
views
Training embedding model on entire corpus then classifying documents from that corpus using trained embeddings
Let's say I have a large corpus of documents. Instead of using a pretrained embedding model, I train my own non-contextual embedding model like w2v/fasttext from scratch on all the documents and save ...
- 209
1
vote
1
answer
499
views
What are the probabilities in the embedding layer of a Word2Vec?
I am trying to understand how a Word2Vec is being trained. I understand that it can be trained using a CBOW and SkipGram. I am however lost as to what the probabilities are in the embedding layer.
...
- 329
1
vote
1
answer
164
views
Does negative sampling sacrifice performance?
I am implementing a Seq2Seq model. Each step of the decoder has |N| outputs (the number of unique words). Since |N| is huge, I am trying to speed up the training by negative sampling (n_sample=100). I ...
- 2,095
5
votes
1
answer
1k
views
glove: king - man + woman != queen
I downloaded 'glove.twitter.27B.25d.txt' from here https://nlp.stanford.edu/projects/glove/, and out of curiosity I wanted to see if king - man + woman does indeed ...
- 2,721
1
vote
0
answers
25
views
Word2Vec Skip Gram Concept/Training Clarifications
I’m implementing the Skip Gram Model from scratch for a project of mine but have a few questions I need cleared up to get the full understanding:
1) What is the size of the output layer? I’m getting ...
- 11
1
vote
1
answer
312
views
Skip-gram model multiplicative constant in the objective function?
I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $...
- 271
1
vote
2
answers
471
views
Having more feed forward layers as hidden layers in word2vec
Word2vec only have one hidden layer followed by a softmax layer. If we add more hidden layer(fully connected feed forward layers), then the model complexity is increased and likely we will get a more ...
- 31