Questions tagged [word2vec]
Word2vec is a neural network that represents words as vectors in a high dimensional space.
208
questions
2
votes
0
answers
30
views
Example orthonormal basis of Word Embedding Space?
Models such as Word2Vec supposedly provide a bijection between language tokens and some "latent-space" that is in fact a high-dimensional vector space.
If this is a vector space, it should ...
- 21
0
votes
0
answers
38
views
How does the chain-rule look for the gradient of a loss function?
When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like:
$...
- 1
1
vote
0
answers
19
views
Team vectors similar to word vectors for a corpus of text
Is there any way I could iteratively create a set of vectors, similar to vectors when embedding words through word2vec, that could represent vectors between teams, and also capture information about a ...
- 11
0
votes
0
answers
14
views
How are weights adjusted iteratively in Doc2Vec neural network and how are results from different word predictions combined?
For those unfamiliar, Doc2Vec is referred to as a 'simple' neural network with one hidden layer that is built very similar to word2vec. As I understand it (in dbow, the implementation I am using), the ...
- 31
2
votes
0
answers
48
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 66
1
vote
0
answers
159
views
When training word2vec, why is a new negative sampling process formulated instead of simply downsampling?
(For background, see The Skip-Gram Model.1 This question does not exactly use their notation, but you should be able to follow along.)
The original skip-gram log-likelihood of a single word, $w$, ...
- 1,430
0
votes
1
answer
36
views
Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?
In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
- 31
1
vote
1
answer
106
views
How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?
I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
- 31
1
vote
1
answer
201
views
How does training word embeddings bring similar words closer together?
How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
- 485
1
vote
1
answer
356
views
Why the Transformer model does not require negative sampling but word2vec does?
Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side.
For word2vec models, negative sampling is used for computational reasons:
Is negative sampling ...
- 2,475
1
vote
0
answers
54
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 11
0
votes
1
answer
326
views
Why should the weight matrix encode word embeddings in CBOW/skip-gram?
Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings.
I'm looking for an intuitive ...
1
vote
0
answers
210
views
Skip gram model and negative log loss likelihood
I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
- 23
2
votes
1
answer
57
views
Text similarity for badly written text
Consider the following scenario:
Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
- 337
2
votes
1
answer
623
views
Should I use cosine or dot similarity inside word2vec's neural network?
I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
- 185
1
vote
1
answer
330
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
- 41
0
votes
0
answers
40
views
Continuous Bag of Words NY Time Corpus
I am working to implement the continuous bag of words approach on the New York Times corpus dataset. However, I am getting word embeddings that do not seem very useful based on a few examples of ...
- 51
0
votes
1
answer
193
views
What is a better way to classify text using word2vec?
I am using word2vec to classify documents into various categories. Let's say we have a document:
Thousands of people with student loan debt will have their debt ...
- 101
1
vote
0
answers
146
views
Using doc2vec embeddings as model input our perhaps similarity comparison? [closed]
Doc2vec is an extension of word2vec, which creates vector representations of documents.
One can use this representations as input to some classifier/regression(Logistic Regression, XGboost, LightGBM .....
- 251
1
vote
2
answers
651
views
What makes training time longer with bigger parameter size in a deep learning model?
I try to understand, is it always the case with the more parameter you trained, the more training time you need when training a deep learning model. For example, i have a CNN model for text ...
- 121
1
vote
1
answer
102
views
Ensure trained word embeddings get high similarity with particular words
I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times
...
- 1,087
1
vote
1
answer
619
views
Does not 'removing the stopwords' affect Natural Language Processing Results in a high degree?
Most stopword lists contain contradicting prepositions (before-after, into-out of) and negativity words (not, no). Removing such words from the text almost always changes the meaning drastically.
The ...
- 129
0
votes
1
answer
527
views
Word2vec/SkipGram: Why softmax?
In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling.
But if in the ...
- 103
0
votes
0
answers
78
views
How the Negative Sampling algorithm chooses the negative samples (k) in character-based embeddings for the Word2Vec model?
In the context of word-based embeddings, the Negative Sampling algorithm chooses negative samples (k) from the most frequent words in the corpora which usually present less meaningful information than ...
MManahi
1
vote
0
answers
523
views
Skip-gram gradient with respect to outer word vector
For the skip gram model we know the naive softmax loss is as follows;-
$$ J=-\log(p(o \mid c)) = -\log\left(\frac {\exp\left(u_{o}^{T} v_{c}\right)}{\sum_{w=1}^{W}\exp\left(u_{w}^{T} v_{c}\right)}\...
- 131
0
votes
0
answers
29
views
Word-embedding does not provide expected relations between words
I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
2
votes
1
answer
409
views
How does Word2Vec CBOW softmax work with multiple context words?
I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4):
$$
p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
- 123
1
vote
0
answers
300
views
Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?
I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer).
However, ...
- 6,996
7
votes
1
answer
2k
views
Do we use maximum likelihood or cross entropy Loss for training skip-gram model?
In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where
$$J(\theta) = -\frac{1}{T}\sum_{t=1}^...
- 1,116
1
vote
1
answer
279
views
Extracting word embedding features of a sentence using Transformer-XL
As you know, there are several pre-trained models that we can use to extract word embeddings.
As an example, I can use the following codes to retrieve word2vec features of my text:
...
- 395
2
votes
1
answer
2k
views
Distance measure in word2vec
I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation.
I have often read that the ...
- 131
1
vote
1
answer
318
views
Intuition for GloVe word embeddings
I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used.
The ...
- 1,116
4
votes
1
answer
774
views
Why is the softmax on the dot product of the word embedding is the probability of context given word?
I was learning about the Word2Vec model, and the following equation was shown:
$\huge{p(o|c) = \frac{exp(u^T_ov_c)}{\sum_{w\in{V}}exp(u^T_wv_c)}}$
in words, the probability of the context word given ...
- 263
4
votes
1
answer
925
views
Word2Vec vs. Doc2Vec Word Vectors
I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access.
My ...
- 1,582
1
vote
0
answers
98
views
Can I skip the Keras Embedding Layer if I already transformed the data to Word2Vec (Google News 300 format)?
Trying to do sentiment analysis with an LSTM NN. I think I understand what the embedding layer does: map each word to a fixed-di-vector.
However, previously, for each text sample, I transformed each ...
- 11
0
votes
1
answer
281
views
When to use documents vs. sentences for Word2Vec?
I have a collection of words from different communities. Each community has a different way of using language and will provide a different word embedding. I can concatenate the sentences from the ...
- 3
2
votes
1
answer
95
views
Debiasing word embeddings
I'm reading the paper titled "man to computer programmer is woman to homemaker. Debiasing word embeddings". I'm right now trying to figure out the math and logic behind it and was doing OK ...
- 153
4
votes
1
answer
457
views
Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?
I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
- 1,007
1
vote
0
answers
114
views
NLP for customer reviews and summaries
I'm trying to develop a model in R that will compare a customer review with a summary of that review that is completed by an employee. The purpose is to ensure that the employee is accurately tagging ...
- 11
3
votes
1
answer
1k
views
Word embeddings - Pre-trained tokenizers vs more involved methods
I'm drowning under all the various methods of converting my text corpora into embeddings.
I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using ...
- 205
2
votes
0
answers
631
views
What is GloVe model's loss function actually minimizing?
I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
- 1,590
2
votes
3
answers
434
views
How can "word hashing" cause a collision in DSSM?
They say in their paper, that "word hashing" can cause a collision. But I don't understand, how. For example, if word good is tranformed to ...
- 412
1
vote
1
answer
1k
views
Can Latent Dirichlet Allocation (LDA) be used to generate word embeddings?
In the original Word2Vec paper (Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013), I came across this phrase:
Many different types of models were proposed for ...
- 113
2
votes
1
answer
1k
views
Generating Sentence Vectors from Word2Vec
I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec.
I read lot of materials and found that Averaging the ...
- 131
1
vote
1
answer
176
views
Training embedding model on entire corpus then classifying documents from that corpus using trained embeddings
Let's say I have a large corpus of documents. Instead of using a pretrained embedding model, I train my own non-contextual embedding model like w2v/fasttext from scratch on all the documents and save ...
- 209
1
vote
1
answer
478
views
What are the probabilities in the embedding layer of a Word2Vec?
I am trying to understand how a Word2Vec is being trained. I understand that it can be trained using a CBOW and SkipGram. I am however lost as to what the probabilities are in the embedding layer.
...
- 329
1
vote
1
answer
158
views
Does negative sampling sacrifice performance?
I am implementing a Seq2Seq model. Each step of the decoder has |N| outputs (the number of unique words). Since |N| is huge, I am trying to speed up the training by negative sampling (n_sample=100). I ...
- 2,095
5
votes
1
answer
1k
views
glove: king - man + woman != queen
I downloaded 'glove.twitter.27B.25d.txt' from here https://nlp.stanford.edu/projects/glove/, and out of curiosity I wanted to see if king - man + woman does indeed ...
- 2,671
1
vote
0
answers
25
views
Word2Vec Skip Gram Concept/Training Clarifications
I’m implementing the Skip Gram Model from scratch for a project of mine but have a few questions I need cleared up to get the full understanding:
1) What is the size of the output layer? I’m getting ...
- 11
1
vote
1
answer
306
views
Skip-gram model multiplicative constant in the objective function?
I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $...
- 271