All Questions
Tagged with word-embeddings word2vec
115 questions
1
vote
0
answers
20
views
Team vectors similar to word vectors for a corpus of text
Is there any way I could iteratively create a set of vectors, similar to vectors when embedding words through word2vec, that could represent vectors between teams, and also capture information about a ...
- 11
2
votes
0
answers
50
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 66
0
votes
1
answer
44
views
Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?
In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
- 31
1
vote
1
answer
120
views
How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?
I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
- 31
1
vote
1
answer
226
views
How does training word embeddings bring similar words closer together?
How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
- 485
1
vote
0
answers
57
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 11
0
votes
1
answer
336
views
Why should the weight matrix encode word embeddings in CBOW/skip-gram?
Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings.
I'm looking for an intuitive ...
1
vote
0
answers
229
views
Skip gram model and negative log loss likelihood
I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
- 23
2
votes
1
answer
58
views
Text similarity for badly written text
Consider the following scenario:
Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
- 337
2
votes
1
answer
649
views
Should I use cosine or dot similarity inside word2vec's neural network?
I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
- 185
1
vote
1
answer
340
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
- 41
0
votes
0
answers
47
views
Continuous Bag of Words NY Time Corpus
I am working to implement the continuous bag of words approach on the New York Times corpus dataset. However, I am getting word embeddings that do not seem very useful based on a few examples of ...
- 51
1
vote
2
answers
721
views
What makes training time longer with bigger parameter size in a deep learning model?
I try to understand, is it always the case with the more parameter you trained, the more training time you need when training a deep learning model. For example, i have a CNN model for text ...
- 121
1
vote
1
answer
105
views
Ensure trained word embeddings get high similarity with particular words
I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times
...
- 1,097
1
vote
1
answer
621
views
Does not 'removing the stopwords' affect Natural Language Processing Results in a high degree?
Most stopword lists contain contradicting prepositions (before-after, into-out of) and negativity words (not, no). Removing such words from the text almost always changes the meaning drastically.
The ...
- 129
0
votes
1
answer
546
views
Word2vec/SkipGram: Why softmax?
In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling.
But if in the ...
- 103
0
votes
0
answers
29
views
Word-embedding does not provide expected relations between words
I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
2
votes
1
answer
433
views
How does Word2Vec CBOW softmax work with multiple context words?
I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4):
$$
p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
- 123
1
vote
0
answers
307
views
Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?
I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer).
However, ...
- 7,074
1
vote
1
answer
279
views
Extracting word embedding features of a sentence using Transformer-XL
As you know, there are several pre-trained models that we can use to extract word embeddings.
As an example, I can use the following codes to retrieve word2vec features of my text:
...
- 395
2
votes
1
answer
2k
views
Distance measure in word2vec
I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation.
I have often read that the ...
- 131
1
vote
1
answer
330
views
Intuition for GloVe word embeddings
I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used.
The ...
- 1,116
2
votes
1
answer
95
views
Debiasing word embeddings
I'm reading the paper titled "man to computer programmer is woman to homemaker. Debiasing word embeddings". I'm right now trying to figure out the math and logic behind it and was doing OK ...
- 153
4
votes
1
answer
462
views
Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?
I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
- 1,027
3
votes
1
answer
1k
views
Word embeddings - Pre-trained tokenizers vs more involved methods
I'm drowning under all the various methods of converting my text corpora into embeddings.
I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using ...
- 205
2
votes
0
answers
639
views
What is GloVe model's loss function actually minimizing?
I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
- 1,630
1
vote
1
answer
1k
views
Can Latent Dirichlet Allocation (LDA) be used to generate word embeddings?
In the original Word2Vec paper (Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013), I came across this phrase:
Many different types of models were proposed for ...
- 113
2
votes
1
answer
1k
views
Generating Sentence Vectors from Word2Vec
I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec.
I read lot of materials and found that Averaging the ...
- 131
1
vote
1
answer
177
views
Training embedding model on entire corpus then classifying documents from that corpus using trained embeddings
Let's say I have a large corpus of documents. Instead of using a pretrained embedding model, I train my own non-contextual embedding model like w2v/fasttext from scratch on all the documents and save ...
- 209
1
vote
1
answer
499
views
What are the probabilities in the embedding layer of a Word2Vec?
I am trying to understand how a Word2Vec is being trained. I understand that it can be trained using a CBOW and SkipGram. I am however lost as to what the probabilities are in the embedding layer.
...
- 329
2
votes
1
answer
141
views
How does using the same vector for the center word and for the context word impact the performance of word vectors in word2vec?
By default, word2vec uses 2 vectors for each word: one for the center word and one for the context word:
$\color{steelblue}{\large \text{Word2vec: objective function}}$
$\color{darkred}{\scriptstyle{\...
- 47.6k
0
votes
0
answers
31
views
What Keras models would be best for output of lists of word vectors?
Imagine a regression model that is to be trained on data consisting of questions and answers expressed in text.
The questions and answers are converted to lists of word vectors using some good word ...
- 141
0
votes
0
answers
55
views
How the values in word2vec embedding are created for each word
I was going through word2vec materials from Andrew Ng's course and below is what i understood.
-> Step1
A matrix of shape embedding_size*number_of_unique_words is created and populated with random ...
- 1,124
0
votes
0
answers
37
views
Unable to learn weights of a Word2Vec model [duplicate]
I was going to implement a word embedding model - namely Word2Vec - by following this TensorFlow tutorial and adapting the code a little bit. Unfortunately, though, my model won't learn anything. I've ...
2
votes
0
answers
37
views
How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?
This is related to my earlier question, which I'm trying to break down into parts (this being the first) since it seemed too large. I'm reading notes on word vectors here. Specifically, I'm referring ...
- 750
1
vote
0
answers
22
views
Understanding how continuous bag of words method learns embedded representations
I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding:
If we have a vocabulary $V$, the naive way to ...
- 750
1
vote
1
answer
295
views
How to create word vectors from short sentences having mixed language (English and Hinglish)? [closed]
How do I create word vectors from a corpus where sentences are very short. e.g if the corpus contains messages from users - 'good morning', 'hello!', 'No, I can't.', 'Where?' etc. One way to resolve ...
- 11
1
vote
0
answers
267
views
How can I define a accuracy measure for word2vec predictions
I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
- 111
2
votes
1
answer
339
views
Is negative sampling only used for computational reasons?
Is negative sampling only used for computational reasons in word2vec and other embedding algorithms? Or are there other benefits?
- 23
1
vote
1
answer
279
views
How are word vector representations derived in Skip-gram?
I was reading the paper Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013 NIPS) and came across a part that I cannot quite understand.
Specifically, ...
- 4,177
1
vote
1
answer
1k
views
How is addition and subtraction determined when using word embeddings like Word2Vec?
This question is particularly in the context of the word embedding algorithm Word2Vec.
I've noticed that many examples that are given in the original paper and other blog posts say things like ...
- 4,177
1
vote
1
answer
1k
views
Word2vec Skip-Gram - Overfitting
i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm:
So my model creates each run nearly ...
- 11
3
votes
1
answer
70
views
Is it possible to take a pretrained word embedding, trained on a general vocabulary, and make it domain specific?
Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus ...
- 12.1k
1
vote
1
answer
3k
views
Efficiently normalize word embeddings
I'm using glove word embedding and would like to [-1,1] normalize it using python. The data is in the format of a dict with the word as key and a ...
- 23
4
votes
0
answers
522
views
Can you use VAEs to produce deep word embeddings?
There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
- 203
2
votes
1
answer
2k
views
Should I use pretrained word2vec or train word2vec on my own dataset? [closed]
I am trying to perfrom fake news detection using machine learning naive bayes classifier. So far I have used BOW and TFIDF as my feature vectors. From research I have found that word embeddings plays ...
- 123
1
vote
1
answer
231
views
What is the motivation to train one's own word embedding model?
I've been using a few big word embedding models like word2vec & FastText, and they work very well on most problems. I am now adressing a new kind of data, on which they perform quite poorly, and I ...
0
votes
1
answer
543
views
Which layer is saved by CBOW? [duplicate]
The word2vec model saves its layer weights as embeddings.
But do CBOW and skipgram both store the input layer weights?
I know they learn different embeddings for the words in the context and for the ...
- 1
1
vote
0
answers
26
views
Why does the "window-based" model fail to take advantage of the repetition?
In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says
"Unlike the matrix factorization methods, the shallow window-based
methods suffer from the disadvantage that they do not ...
- 879
0
votes
1
answer
991
views
Negative values in word vectorizations
I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
- 251