Questions tagged [word2vec]
Word2vec is a neural network that represents words as vectors in a high dimensional space.
60 questions with no upvoted or accepted answers
7
votes
0
answers
730
views
Understanding Object2Vec
AWS released an interesting feature as part of the SageMaker service called Object2Vec that lets you make an embedding for search out of pretty much anything: documents, users, products, ...
- 6,767
6
votes
0
answers
2k
views
Why word embeddings learned from word2vec are linearly correlated
I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino ...
- 1,176
4
votes
0
answers
522
views
Can you use VAEs to produce deep word embeddings?
There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
- 203
4
votes
1
answer
356
views
Skipgram - multiple formulations?
I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions.
1 - Taking a look at this blog post and Andrew Ng's Deep Learning Specialization, I understood ...
- 521
4
votes
0
answers
363
views
Why PMI + SVD works for similarities arithmetics?
Recently Julia Silge blogged here and here, quoting blog entry by Chris Moody, who suggested that the similarities arithmetic in word2vec can be approximated by using PMI indexes followed by SVD ...
- 141k
3
votes
1
answer
70
views
Is it possible to take a pretrained word embedding, trained on a general vocabulary, and make it domain specific?
Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus ...
- 12.1k
3
votes
1
answer
1k
views
Does hierarchical softmax of skip gram and CBOW only update output vectors on the path from the root to the actual output word?
After reading word2vec Parameter Learning Explained by Xin Rong, I understand that in the hierarchical softmax model, there is no output vector representation for words, instead, each of the $V-1$ ...
- 620
2
votes
0
answers
35
views
Example orthonormal basis of Word Embedding Space?
Models such as Word2Vec supposedly provide a bijection between language tokens and some "latent-space" that is in fact a high-dimensional vector space.
If this is a vector space, it should ...
- 21
2
votes
0
answers
50
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 66
2
votes
0
answers
639
views
What is GloVe model's loss function actually minimizing?
I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
- 1,630
2
votes
1
answer
141
views
How does using the same vector for the center word and for the context word impact the performance of word vectors in word2vec?
By default, word2vec uses 2 vectors for each word: one for the center word and one for the context word:
$\color{steelblue}{\large \text{Word2vec: objective function}}$
$\color{darkred}{\scriptstyle{\...
- 47.6k
2
votes
0
answers
37
views
How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?
This is related to my earlier question, which I'm trying to break down into parts (this being the first) since it seemed too large. I'm reading notes on word vectors here. Specifically, I'm referring ...
- 750
2
votes
0
answers
363
views
Could we drop the hidden layer in a skip-gram word2vec and train only a square weight matrix?
After pondering on the (skip-gram) word2vec algorithm and the fact that its single hidden layer is linearly activated, I am not 100% sure that I understand the significance of everything that is ...
- 21
2
votes
0
answers
94
views
Machine Learning: mathematical verification of this text-to-image cross entropy loss function?
I'm implementing a research paper on GANs and have come across this rather convoluted text-image loss function which has these main components:
$$P(D_i | Q_i) = \frac{\exp({\gamma_3 R(Q_i, D_i)})}{\...
- 1,783
2
votes
0
answers
312
views
Why not detect convergence in word2vec (skip-gram and cbow)?
In the word2vec software, as well as the implementation in gensim, training is done for a given number of epochs, and the learning rate (alpha) is decreased every 10000 words till a minimal alpha. ...
- 21
2
votes
1
answer
690
views
Find most similar sentence from one list of sentences to another
I have two lists of short sentences (List A and List B). For each short sentence in List A, I am trying to find the most similar short sentence in List B.
Each list has a different count of elements ...
- 21
2
votes
1
answer
544
views
How should I formalize Doc2Vec Matrix Dimension?
Below, I have a simple diagram explaining the matrix dimension of word2vec.
My goal is to expand this graph to incorporate document vectors for doc2vec. However, I'm having trouble understanding the ...
- 193
2
votes
0
answers
86
views
Are there good algorithms for finding sentences given only their sentence embedding?
A common baseline that often works really well for sentence vectors is simply to average the word vectors for each word in the sentence. So lets say I have a vector made in this way. If I have a ...
- 2,945
2
votes
0
answers
213
views
Overview research on ambiguity of words
As far as I know vector representations of word embeddings do not account for ambiguity. A single word can have different meanings f.e. "hot" can mean very good looking or very warm. Another example ...
- 151
2
votes
0
answers
2k
views
Text classification with word2vec and neural nets [spacy.io, keras]
I have about 300.000 messages with bodies and titles at hand.
~20% are of them labeled positive.
Right now, I run the word2vec feature generation with spacy.io (excellent library btw.), generating ...
- 151
1
vote
0
answers
20
views
Team vectors similar to word vectors for a corpus of text
Is there any way I could iteratively create a set of vectors, similar to vectors when embedding words through word2vec, that could represent vectors between teams, and also capture information about a ...
- 11
1
vote
0
answers
173
views
When training word2vec, why is a new negative sampling process formulated instead of simply downsampling?
(For background, see The Skip-Gram Model.1 This question does not exactly use their notation, but you should be able to follow along.)
The original skip-gram log-likelihood of a single word, $w$, ...
- 1,625
1
vote
0
answers
57
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 11
1
vote
0
answers
229
views
Skip gram model and negative log loss likelihood
I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
- 23
1
vote
0
answers
536
views
Skip-gram gradient with respect to outer word vector
For the skip gram model we know the naive softmax loss is as follows;-
$$ J=-\log(p(o \mid c)) = -\log\left(\frac {\exp\left(u_{o}^{T} v_{c}\right)}{\sum_{w=1}^{W}\exp\left(u_{w}^{T} v_{c}\right)}\...
- 131
1
vote
0
answers
307
views
Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?
I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer).
However, ...
- 7,074
1
vote
0
answers
99
views
Can I skip the Keras Embedding Layer if I already transformed the data to Word2Vec (Google News 300 format)?
Trying to do sentiment analysis with an LSTM NN. I think I understand what the embedding layer does: map each word to a fixed-di-vector.
However, previously, for each text sample, I transformed each ...
- 11
1
vote
0
answers
117
views
NLP for customer reviews and summaries
I'm trying to develop a model in R that will compare a customer review with a summary of that review that is completed by an employee. The purpose is to ensure that the employee is accurately tagging ...
- 11
1
vote
1
answer
177
views
Training embedding model on entire corpus then classifying documents from that corpus using trained embeddings
Let's say I have a large corpus of documents. Instead of using a pretrained embedding model, I train my own non-contextual embedding model like w2v/fasttext from scratch on all the documents and save ...
- 209
1
vote
1
answer
164
views
Does negative sampling sacrifice performance?
I am implementing a Seq2Seq model. Each step of the decoder has |N| outputs (the number of unique words). Since |N| is huge, I am trying to speed up the training by negative sampling (n_sample=100). I ...
- 2,095
1
vote
0
answers
25
views
Word2Vec Skip Gram Concept/Training Clarifications
I’m implementing the Skip Gram Model from scratch for a project of mine but have a few questions I need cleared up to get the full understanding:
1) What is the size of the output layer? I’m getting ...
- 11
1
vote
0
answers
24
views
Detecting anomalies in system logs
Chances are this may be closed off as too broad, but I'll try to be as specific as I can.
I am currently working with API logs with categorical features separated by 1ms intervals, an example:
...
- 111
1
vote
0
answers
22
views
Understanding how continuous bag of words method learns embedded representations
I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding:
If we have a vocabulary $V$, the naive way to ...
- 750
1
vote
0
answers
641
views
In Skip-Gram model, what is the intuition of taking average of input word matrix and output word matrix?
In the skip-gram model, we have input word matrix $W$, and output word matrix $W'$.
The final word embedding we got, is the average of $W$ and $W'$.
what is the intuition of taking average of $W$ and $...
- 121
1
vote
0
answers
267
views
How can I define a accuracy measure for word2vec predictions
I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
- 111
1
vote
1
answer
50
views
Word coocurrence in word2vec
Based on the vector values of two words in word2vec, could we judge whether they co-occur and frequencies of coocurrence?
- 1,087
1
vote
0
answers
300
views
Finding similarity dissimilarity between different groups of vectors
Suppose I need to combine or group together set of vectors in one area and another group in other areas, however I need to place these groups in a plot so they are scattered in the screen per ...
- 11
1
vote
0
answers
26
views
Why does the "window-based" model fail to take advantage of the repetition?
In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says
"Unlike the matrix factorization methods, the shallow window-based
methods suffer from the disadvantage that they do not ...
- 879
1
vote
0
answers
1k
views
What is the optimal sampling rate / window size for Word2Vec Continuous Bag of Words?
What is the greatest number of embeddings you can average for the Word2Vec Cbow algorithm before measures of quality start dropping?
For skip-gram I've seen window sizes up to 20 work, but I imagine ...
- 1,229
1
vote
2
answers
1k
views
Anomaly detection in Text Classification
I have built a text classifier using OneClassSVM.
I have the training set which corresponds to only one label i.e("Yes") and I don't have the other("NO") label data.
My task is to build a classifier ...
- 11
1
vote
0
answers
125
views
Is there a pre-trained word embedding for english song lyrics?
I'm working on a project where the dataset is English songs. So I need word embeddings which are trained on English songs.
If none exist, Could you please suggest one that matches for this use case?
- 111
1
vote
0
answers
170
views
Great ways to identify adult content in text
What are some good ways to identify adult content in text. It is definitely a text classification problem, but how do we handle words that are spelt like @$$.
- 11
1
vote
1
answer
377
views
Correctness of a skewed cosine similarity graph
I am currently implementing a word2vec model that uses the cosine similarity to determine the similarity between two vectors.
When plotting all the possible cosine similarities, I get the following ...
- 173
1
vote
2
answers
3k
views
Learning image embeddings using VGG and Word2Vec
Background: In word2vec we pass in a one-hot encoding of our target word into a simple neural network which is trained to predict context words from a window around our target. We eventually take the ...
- 291
1
vote
0
answers
161
views
Improving skip gram by transforming and reducing number of training examples
in skip-gram model we have for arbitrary value of k (window size) and every word occurence in corpus k-1 training examples, what if we could take each word, count output words occurences, transform it ...
- 498
1
vote
0
answers
924
views
Word embeddings through SVD - Why are the singular values thrown away?
Word embeddings are created from matrix factorization of the word- word co-occurrences matrix X through SVD.
So if we factorize X = USV and if we are taking only k significant singular values into ...
1
vote
0
answers
143
views
What is the benefit of word embedding wrapper over simply adding an extra layer in RNNs
I am trying to build a sequence to sequence model using tensorflow. Tensorflow provides some functions for learning embeddings for words - tf.nn.embedding_lookup and tf.contrib.rnn.EmbeddingWrapper ...
1
vote
0
answers
161
views
How to show mathematically that probability distribution of a word vector is better than another
I an newbie to the statistics and probability. I have been working on Latent Dirichlet Allocation - Topic Coherence Optimization. As a part of my research work, I have two word vectors, say V1 and V2. ...
- 11
1
vote
0
answers
167
views
Embedding Phrase using Recurrent Neural Network
I am trying to get a word2vec embedding of a given phrase using an LSTM RNN encoder.
For example, "How are you?" => [some word2vec representation]
I have pretrained model of word2vec(i already wrote ...
- 123
0
votes
0
answers
40
views
How does the chain-rule look for the gradient of a loss function?
When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like:
$...
- 1