Questions tagged [word2vec]

Word2vec is a neural network that represents words as vectors in a high dimensional space.

Filter by
Sorted by
Tagged with
0 votes
0 answers
31 views

How does the chain-rule look for the gradient of a loss function?

When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like: $...
ZenPyro's user avatar
1 vote
0 answers
19 views

Team vectors similar to word vectors for a corpus of text

Is there any way I could iteratively create a set of vectors, similar to vectors when embedding words through word2vec, that could represent vectors between teams, and also capture information about a ...
troy3.14's user avatar
0 votes
0 answers
13 views

How are weights adjusted iteratively in Doc2Vec neural network and how are results from different word predictions combined?

For those unfamiliar, Doc2Vec is referred to as a 'simple' neural network with one hidden layer that is built very similar to word2vec. As I understand it (in dbow, the implementation I am using), the ...
osckt's user avatar
  • 31
2 votes
0 answers
47 views

How was the word2vec model trained?

Let's take the CBOW (continuous bag of words) model as the example. Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
J. Doe's user avatar
  • 66
1 vote
0 answers
122 views

When training word2vec, why is a new negative sampling process formulated instead of simply downsampling?

(For background, see The Skip-Gram Model.1 This question does not exactly use their notation, but you should be able to follow along.) The original skip-gram log-likelihood of a single word, $w$, ...
chicxulub's user avatar
  • 1,400
0 votes
0 answers
25 views

How does Doc2Vec handle documents where the length is less than the window size?

I am using doc2vec to vectorise documents that average in length 10 words (PV-DBOW implementation of the algorithm). I am wondering how doc2vec handles cases where the number of words in a document is ...
osckt's user avatar
  • 31
0 votes
0 answers
18 views

How do Doc2Vec PV-DBOW iterations/epoch's work?

Please correct me if I'm wrong but as I understand it, one iteration (?) of doc2vec takes each document and predicts a series of randomly sampled context words individually, feeding the errors from ...
osckt's user avatar
  • 31
0 votes
1 answer
27 views

Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?

In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
osckt's user avatar
  • 31
1 vote
1 answer
73 views

How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?

I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
osckt's user avatar
  • 31
0 votes
1 answer
124 views

How does training word embeddings bring similar words closer together?

How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
Glue's user avatar
  • 463
0 votes
0 answers
31 views

Word2Vec with some Levenshtein metric

I'm building a search engine of technical documents based on Word2Vec, using cosine similarity metric. This search engine is very specific because it is meant to work with technical writings written ...
Aurélien Pierre's user avatar
1 vote
1 answer
242 views

Why the Transformer model does not require negative sampling but word2vec does?

Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side. For word2vec models, negative sampling is used for computational reasons: Is negative sampling ...
CyberPlayerOne's user avatar
1 vote
0 answers
49 views

How does softmax work for vectors?

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
Ruediger's user avatar
0 votes
0 answers
42 views

Training a text model for similarity

I'm training a text similarity model using a two tower approach. The data I'm dealing with has a lot of unusual words that are important (names of people, places) that also don't appear in any pre-...
mmmmo's user avatar
  • 111
0 votes
1 answer
287 views

Why should the weight matrix encode word embeddings in CBOW/skip-gram?

Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings. I'm looking for an intuitive ...
Michael Ray's user avatar
1 vote
0 answers
166 views

Skip gram model and negative log loss likelihood

I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
MrMercury's user avatar
2 votes
1 answer
54 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
Ramiro Hum-Sah's user avatar
2 votes
1 answer
530 views

Should I use cosine or dot similarity inside word2vec's neural network?

I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
oliver.c's user avatar
1 vote
1 answer
279 views

Feedforward Neural Net Language Model - computational complexity (word2vec)

While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM): It consists of input, projection, hidden and output layers. At the ...
MDescamps's user avatar
0 votes
0 answers
37 views

Continuous Bag of Words NY Time Corpus

I am working to implement the continuous bag of words approach on the New York Times corpus dataset. However, I am getting word embeddings that do not seem very useful based on a few examples of ...
dzheng1887's user avatar
0 votes
1 answer
186 views

What is a better way to classify text using word2vec?

I am using word2vec to classify documents into various categories. Let's say we have a document: Thousands of people with student loan debt will have their debt ...
suhdonghwi's user avatar
1 vote
0 answers
133 views

Using doc2vec embeddings as model input our perhaps similarity comparison? [closed]

Doc2vec is an extension of word2vec, which creates vector representations of documents. One can use this representations as input to some classifier/regression(Logistic Regression, XGboost, LightGBM .....
Borut Flis's user avatar
1 vote
2 answers
453 views

What makes training time longer with bigger parameter size in a deep learning model?

I try to understand, is it always the case with the more parameter you trained, the more training time you need when training a deep learning model. For example, i have a CNN model for text ...
thenoirlatte's user avatar
1 vote
1 answer
101 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...
Clock Slave's user avatar
  • 1,067
1 vote
1 answer
606 views

Does not 'removing the stopwords' affect Natural Language Processing Results in a high degree?

Most stopword lists contain contradicting prepositions (before-after, into-out of) and negativity words (not, no). Removing such words from the text almost always changes the meaning drastically. The ...
Xfce4's user avatar
  • 129
0 votes
1 answer
480 views

Word2vec/SkipGram: Why softmax?

In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling. But if in the ...
guest's user avatar
  • 103
0 votes
0 answers
75 views

How the Negative Sampling algorithm chooses the negative samples (k) in character-based embeddings for the Word2Vec model?

In the context of word-based embeddings, the Negative Sampling algorithm chooses negative samples (k) from the most frequent words in the corpora which usually present less meaningful information than ...
user avatar
1 vote
0 answers
477 views

Skip-gram gradient with respect to outer word vector

For the skip gram model we know the naive softmax loss is as follows;- $$ J=-\log(p(o \mid c)) = -\log\left(\frac {\exp\left(u_{o}^{T} v_{c}\right)}{\sum_{w=1}^{W}\exp\left(u_{w}^{T} v_{c}\right)}\...
Divyaanand Sinha's user avatar
0 votes
0 answers
29 views

Word-embedding does not provide expected relations between words

I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
user320342's user avatar
2 votes
1 answer
359 views

How does Word2Vec CBOW softmax work with multiple context words?

I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4): $$ p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
jonas's user avatar
  • 123
1 vote
0 answers
280 views

Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?

I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer). However, ...
Charlie Parker's user avatar
8 votes
1 answer
1k views

Do we use maximum likelihood or cross entropy Loss for training skip-gram model?

In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where $$J(\theta) = -\frac{1}{T}\sum_{t=1}^...
calveeen's user avatar
  • 1,126
1 vote
1 answer
267 views

Extracting word embedding features of a sentence using Transformer-XL

As you know, there are several pre-trained models that we can use to extract word embeddings. As an example, I can use the following codes to retrieve word2vec features of my text: ...
Kadaj13's user avatar
  • 395
2 votes
1 answer
2k views

Distance measure in word2vec

I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation. I have often read that the ...
Josef's user avatar
  • 131
1 vote
1 answer
294 views

Intuition for GloVe word embeddings

I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used. The ...
calveeen's user avatar
  • 1,126
3 votes
1 answer
687 views

Why is the softmax on the dot product of the word embedding is the probability of context given word?

I was learning about the Word2Vec model, and the following equation was shown: $\huge{p(o|c) = \frac{exp(u^T_ov_c)}{\sum_{w\in{V}}exp(u^T_wv_c)}}$ in words, the probability of the context word given ...
Moran Reznik's user avatar
4 votes
1 answer
885 views

Word2Vec vs. Doc2Vec Word Vectors

I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access. My ...
Tylerr's user avatar
  • 1,532
1 vote
0 answers
98 views

Can I skip the Keras Embedding Layer if I already transformed the data to Word2Vec (Google News 300 format)?

Trying to do sentiment analysis with an LSTM NN. I think I understand what the embedding layer does: map each word to a fixed-di-vector. However, previously, for each text sample, I transformed each ...
FatTiger's user avatar
0 votes
1 answer
252 views

When to use documents vs. sentences for Word2Vec?

I have a collection of words from different communities. Each community has a different way of using language and will provide a different word embedding. I can concatenate the sentences from the ...
VminVsky's user avatar
2 votes
1 answer
90 views

Debiasing word embeddings

I'm reading the paper titled "man to computer programmer is woman to homemaker. Debiasing word embeddings". I'm right now trying to figure out the math and logic behind it and was doing OK ...
mitra mirshafiee's user avatar
4 votes
1 answer
437 views

Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?

I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
etang's user avatar
  • 907
1 vote
0 answers
109 views

NLP for customer reviews and summaries

I'm trying to develop a model in R that will compare a customer review with a summary of that review that is completed by an employee. The purpose is to ensure that the employee is accurately tagging ...
marigato's user avatar
3 votes
1 answer
1k views

Word embeddings - Pre-trained tokenizers vs more involved methods

I'm drowning under all the various methods of converting my text corpora into embeddings. I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using ...
ML_Engine's user avatar
  • 205
2 votes
0 answers
615 views

What is GloVe model's loss function actually minimizing?

I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
Fraïssé's user avatar
  • 1,520
2 votes
3 answers
430 views

How can "word hashing" cause a collision in DSSM?

They say in their paper, that "word hashing" can cause a collision. But I don't understand, how. For example, if word good is tranformed to ...
Dims's user avatar
  • 402
1 vote
1 answer
1k views

Can Latent Dirichlet Allocation (LDA) be used to generate word embeddings?

In the original Word2Vec paper (Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013), I came across this phrase: Many different types of models were proposed for ...
NeuronQ's user avatar
  • 113
2 votes
1 answer
1k views

Generating Sentence Vectors from Word2Vec

I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec. I read lot of materials and found that Averaging the ...
chaitanya's user avatar
0 votes
1 answer
170 views

Training embedding model on entire corpus then classifying documents from that corpus using trained embeddings

Let's say I have a large corpus of documents. Instead of using a pretrained embedding model, I train my own non-contextual embedding model like w2v/fasttext from scratch on all the documents and save ...
HMK's user avatar
  • 179
1 vote
1 answer
430 views

What are the probabilities in the embedding layer of a Word2Vec?

I am trying to understand how a Word2Vec is being trained. I understand that it can be trained using a CBOW and SkipGram. I am however lost as to what the probabilities are in the embedding layer. ...
Tibo Geysen's user avatar
1 vote
1 answer
142 views

Does negative sampling sacrifice performance?

I am implementing a Seq2Seq model. Each step of the decoder has |N| outputs (the number of unique words). Since |N| is huge, I am trying to speed up the training by negative sampling (n_sample=100). I ...
Munichong's user avatar
  • 2,085

1
2 3 4 5