Questions tagged [word2vec]

Word2vec is a neural network that represents words as vectors in a high dimensional space.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

Is GloVe really learn co-occurrence probabilities ratio (Pik/Pjk in paper) rather than the probabilities themselves?

In the cost function, there is only the co-occurrence, not even co-occurrence probabilities.
user avatar
  • 1
1 vote
1 answer
25 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
user avatar
1 vote
1 answer
31 views

Should I use cosine or dot similarity inside word2vec's neural network?

I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
user avatar
0 votes
0 answers
49 views

Feature importance inference with word2vec in a classification task

I have a binary classification task for tweets in which I am currently testing several models. Surprinsingly, the model that outbeated state of the art algorithms ,such as BERT or BERTweet, is a ...
user avatar
  • 43
0 votes
1 answer
39 views

Feedforward Neural Net Language Model - computational complexity (word2vec)

While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM): It consists of input, projection, hidden and output layers. At the ...
user avatar
0 votes
0 answers
13 views

What is considered as small or large dataset for word2vec?

In the paper "Distributed Representations of Words and Phrases and their Compositionality" that introduced negative sampling (among other things), authors describe the recommended number of ...
user avatar
0 votes
0 answers
15 views

Word embedding to support OOV words for identity embedding

I want to create a model that performs a user-id embedding (hash of a user) for a Graph Neural Network learning task, the problem I am facing is that I might have a very large corpus of users, which ...
user avatar
0 votes
0 answers
14 views

What's the meaning of the notation E in negative sampling objective function?

The negative sampling objective function is like this: I know it's similar to the word2vec's negative sampling objective function. Pn is the negative sampling distribution. But what's the meaning of ...
user avatar
0 votes
0 answers
27 views

Continuous Bag of Words NY Time Corpus

I am working to implement the continuous bag of words approach on the New York Times corpus dataset. However, I am getting word embeddings that do not seem very useful based on a few examples of ...
user avatar
0 votes
1 answer
53 views

What is a better way to classify text using word2vec?

I am using word2vec to classify documents into various categories. Let's say we have a document: Thousands of people with student loan debt will have their debt ...
user avatar
0 votes
0 answers
41 views

Given embedding vector A and vector B, how to find top k embedding vectors such that they are similar to vector A and dissimilar to vector B

Which would be better approach for getting top k embedding vectors such that they are similar to embedding vector A and dissimilar to vector B. Approach 1: calculate ...
user avatar
1 vote
0 answers
40 views

Using doc2vec embeddings as model input our perhaps similarity comparison? [closed]

Doc2vec is an extension of word2vec, which creates vector representations of documents. One can use this representations as input to some classifier/regression(Logistic Regression, XGboost, LightGBM .....
user avatar
0 votes
0 answers
54 views

Plotting cosine similarities in 3 dimensional space made from word embeddings

I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated as follows. First, a ...
user avatar
1 vote
1 answer
27 views

What makes training time longer with bigger parameter size in a deep learning model?

I try to understand, is it always the case with the more parameter you trained, the more training time you need when training a deep learning model. For example, i have a CNN model for text ...
user avatar
0 votes
0 answers
15 views

How is the context matrix updated in CBOW?

If we use softmax activation function, and the error as gradient of softmax wrt the representation of the 'word', only the word representation gets updated. How is the context matrix getting updated? $...
user avatar
  • 1
0 votes
1 answer
32 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...
user avatar
1 vote
1 answer
244 views

Does not 'removing the stopwords' affect Natural Language Processing Results in a high degree?

Most stopword lists contain contradicting prepositions (before-after, into-out of) and negativity words (not, no). Removing such words from the text almost always changes the meaning drastically. The ...
user avatar
  • 129
0 votes
1 answer
135 views

Word2vec/SkipGram: Why softmax?

In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling. But if in the ...
user avatar
  • 3
0 votes
0 answers
47 views

How the Negative Sampling algorithm chooses the negative samples (k) in character-based embeddings for the Word2Vec model?

In the context of word-based embeddings, the Negative Sampling algorithm chooses negative samples (k) from the most frequent words in the corpora which usually present less meaningful information than ...
user avatar
1 vote
0 answers
183 views

Skip-gram gradient with respect to outer word vector

For the skip gram model we know the naive softmax loss is as follows;- $$ J=-\log(p(o \mid c)) = -\log\left(\frac {\exp\left(u_{o}^{T} v_{c}\right)}{\sum_{w=1}^{W}\exp\left(u_{w}^{T} v_{c}\right)}\...
user avatar
0 votes
0 answers
23 views

Word-embedding does not provide expected relations between words

I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
user avatar
1 vote
1 answer
138 views

How does Word2Vec CBOW softmax work with multiple context words?

I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4): $$ p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
user avatar
  • 113
1 vote
0 answers
117 views

Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?

I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer). However, ...
user avatar
4 votes
1 answer
525 views

Do we use maximum likelihood or cross entropy Loss for training skip-gram model?

In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where $$J(\theta) = -\frac{1}{T}\sum_{t=1}^...
user avatar
  • 926
1 vote
1 answer
131 views

Extracting word embedding features of a sentence using Transformer-XL

As you know, there are several pre-trained models that we can use to extract word embeddings. As an example, I can use the following codes to retrieve word2vec features of my text: ...
user avatar
  • 385
2 votes
1 answer
701 views

Distance measure in word2vec

I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation. I have often read that the ...
user avatar
  • 131
1 vote
1 answer
144 views

Intuition for GloVe word embeddings

I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used. The ...
user avatar
  • 926
2 votes
1 answer
250 views

Why is the softmax on the dot product of the word embedding is the probability of context given word?

I was learning about the Word2Vec model, and the following equation was shown: $\huge{p(o|c) = \frac{exp(u^T_ov_c)}{\sum_{w\in{V}}exp(u^T_wv_c)}}$ in words, the probability of the context word given ...
user avatar
4 votes
1 answer
332 views

Word2Vec vs. Doc2Vec Word Vectors

I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access. My ...
user avatar
  • 1,320
1 vote
0 answers
39 views

Can I skip the Keras Embedding Layer if I already transformed the data to Word2Vec (Google News 300 format)?

Trying to do sentiment analysis with an LSTM NN. I think I understand what the embedding layer does: map each word to a fixed-di-vector. However, previously, for each text sample, I transformed each ...
user avatar
0 votes
1 answer
18 views

When to use documents vs. sentences for Word2Vec?

I have a collection of words from different communities. Each community has a different way of using language and will provide a different word embedding. I can concatenate the sentences from the ...
user avatar
2 votes
1 answer
58 views

Debiasing word embeddings

I'm reading the paper titled "man to computer programmer is woman to homemaker. Debiasing word embeddings". I'm right now trying to figure out the math and logic behind it and was doing OK ...
user avatar
4 votes
1 answer
281 views

Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?

I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
user avatar
  • 815
1 vote
0 answers
53 views

NLP for customer reviews and summaries

I'm trying to develop a model in R that will compare a customer review with a summary of that review that is completed by an employee. The purpose is to ensure that the employee is accurately tagging ...
user avatar
  • 11
1 vote
1 answer
951 views

Word embeddings - Pre-trained tokenizers vs more involved methods

I'm drowning under all the various methods of converting my text corpora into embeddings. I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using ...
user avatar
  • 175
0 votes
1 answer
167 views

First two principal components explain 100% variance of tfidf weighted tweet vector data set (300 features)

I am trying to do some analysis on my data set with PCA so I can effectively cluster it with kmeans. My preprocessed data is tokenized, filtered (stopwords, punctuation, etc.), POS tagged, and ...
user avatar
  • 1
1 vote
0 answers
449 views

What is GloVe model's loss function actually minimizing?

I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
user avatar
  • 1,081
2 votes
3 answers
274 views

How can "word hashing" cause a collision in DSSM?

They say in their paper, that "word hashing" can cause a collision. But I don't understand, how. For example, if word good is tranformed to ...
user avatar
  • 402
1 vote
1 answer
594 views

Can Latent Dirichlet Allocation (LDA) be used to generate word embeddings?

In the original Word2Vec paper (Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013), I came across this phrase: Many different types of models were proposed for ...
user avatar
  • 113
2 votes
1 answer
812 views

Generating Sentence Vectors from Word2Vec

I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec. I read lot of materials and found that Averaging the ...
user avatar
0 votes
1 answer
32 views

Training embedding model on entire corpus then classifying documents from that corpus using trained embeddings

Let's say I have a large corpus of documents. Instead of using a pretrained embedding model, I train my own non-contextual embedding model like w2v/fasttext from scratch on all the documents and save ...
user avatar
  • 159
1 vote
1 answer
202 views

What are the probabilities in the embedding layer of a Word2Vec?

I am trying to understand how a Word2Vec is being trained. I understand that it can be trained using a CBOW and SkipGram. I am however lost as to what the probabilities are in the embedding layer. ...
user avatar
0 votes
1 answer
54 views

Does negative sampling sacrifice performance?

I am implementing a Seq2Seq model. Each step of the decoder has |N| outputs (the number of unique words). Since |N| is huge, I am trying to speed up the training by negative sampling (n_sample=100). I ...
user avatar
  • 1,695
3 votes
1 answer
416 views

glove: king - man + woman != queen

I downloaded 'glove.twitter.27B.25d.txt' from here https://nlp.stanford.edu/projects/glove/, and out of curiosity I wanted to see if king - man + woman does indeed ...
user avatar
  • 2,459
1 vote
0 answers
13 views

Word2Vec Skip Gram Concept/Training Clarifications

I’m implementing the Skip Gram Model from scratch for a project of mine but have a few questions I need cleared up to get the full understanding: 1) What is the size of the output layer? I’m getting ...
user avatar
  • 11
1 vote
1 answer
122 views

Skip-gram model multiplicative constant in the objective function?

I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $...
user avatar
  • 261
0 votes
1 answer
166 views

Having more feed forward layers as hidden layers in word2vec

Word2vec only have one hidden layer followed by a softmax layer. If we add more hidden layer(fully connected feed forward layers), then the model complexity is increased and likely we will get a more ...
user avatar
6 votes
0 answers
574 views

Understanding Object2Vec

AWS released an interesting feature as part of the SageMaker service called Object2Vec that lets you make an embedding for search out of pretty much anything: documents, users, products, ...
user avatar
  • 6,047
1 vote
1 answer
262 views

Why CBOW model is called "continuous"?

The question is pretty clear from the Title itself, why the Continuous Bag of Words (CBOW) model is called continuous. I also don't know what exactly "distributed representation" of word vectors ...
user avatar
1 vote
0 answers
22 views

Detecting anomalies in system logs

Chances are this may be closed off as too broad, but I'll try to be as specific as I can. I am currently working with API logs with categorical features separated by 1ms intervals, an example: ...
user avatar

1
2 3 4 5