Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [word2vec]

Word2vec is a neural network that represents words as vectors in a high dimensional space.

0
votes
0answers
7 views

Text generation: Word embedding (vector) prediction instead of softmax probabilities?

0 I'm looking into a project that generates text using LSTM. The common approach to this is having the network output a distribution which you use softmax on to get the most probable word. This ...
0
votes
0answers
11 views

Visualising sentence vectors by averaging word vectors

I have $82114$ sentences for which I have found the vector representation by summing over individual word vectors(using Word2Vec). Now I have a vector representation for each sentence in my dataset. ...
0
votes
0answers
4 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
0answers
21 views

Finding similarity dissimilarity between different groups of vectors

Suppose I need to combine or group together set of vectors in one area and another group in other areas, however I need to place these groups in a plot so they are scattered in the screen per ...
0
votes
0answers
6 views

Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
0
votes
0answers
5 views

contextual embedding algorithm including continuous variables

Is there any work that allows contextual embedding of events that allows not only for categorical information but also some magnitude information (as opposed to word2vec, GloVe etc) I have a series ...
0
votes
1answer
33 views

Understanding word2vec backpropagation

I'm watching the following video on word2vec from University of Waterloo: https://www.youtube.com/watch?v=GMCwS7tS5ZM&t=962s The update function for word my word embedding vector is: $v'_w = v_w ...
0
votes
0answers
18 views

word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
0
votes
0answers
14 views

Clustering positive and negative qualifiers with word2vec

I am looking to find whether a potential qualifier is positive, negative, or unknown. Example positive qualifiers are: increase, positive, raise. Example negative qualifiers are: decrease, negative, ...
0
votes
1answer
53 views

Should I use pretrained word2vec or train word2vec on my own dataset? [closed]

I am trying to perfrom fake news detection using machine learning naive bayes classifier. So far I have used BOW and TFIDF as my feature vectors. From research I have found that word embeddings plays ...
0
votes
0answers
19 views

Word embeddings NLP

Studying about word embeddings I have a few questions because I have been confused. According some textbooks we have two categories of word embeddings the sparse models which based on frequency (word ...
0
votes
0answers
45 views

Learning similarity of representations

I am interested in a framework for mapping together input representations based on some common context. I have looked into word2vec, which does more or less what I want, but I want to know if anyone ...
1
vote
1answer
25 views

Output from Word2Vec

I am working on sentiment analysis. I am using Word2Vec method. I don't understand the output from this code line. ...
0
votes
1answer
14 views

What is the motivation to train one's own word embedding model?

I've been using a few big word embedding models like word2vec & FastText, and they work very well on most problems. I am now adressing a new kind of data, on which they perform quite poorly, and I ...
0
votes
0answers
29 views

Can an embedding layer be replaced by a fully connected layer?

Due to architecture choices and organization of code, I have a file called data.py that processes texts and returns two vectors : X and Y which are the vectorized text and the corresponding label. ...
0
votes
1answer
24 views

Which layer is saved by CBOW?

The word2vec model saves its layer weights as embeddings. But do CBOW and skipgram both store the input layer weights? I know they learn different embeddings for the words in the context and for the ...
1
vote
0answers
6 views

Why does the “window-based” model fail to take advantage of the repetition?

In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says "Unlike the matrix factorization methods, the shallow window-based methods suffer from the disadvantage that they do not ...
1
vote
1answer
59 views

How does Word2Vec ensure that antonyms will be far apart in the vector space

Broadly speaking the training of word2vec is a process in which words that are often in the same context are clustered together in the vector space. We start by randomly shuffling the words on the ...
0
votes
1answer
30 views

Negative values in word vectorizations

I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
1
vote
0answers
20 views

How many word2vec pretrained models are available?

In my experiments with pre-trained word2vec models for NLP tasks, I have so far come across two models - one trained on Google News dataset and another which has been trained on Wikipedia text corpus. ...
0
votes
0answers
15 views

Word2Vec how to handle trend sensitive data from a live feed

(Sorry for the wall of text) TL;DR - I wonder how to regularly retrain a model on trend sensitive data from a live feed. I have been working on developing a Machine Learning model with Word2Vec as ...
0
votes
0answers
73 views

Can recursive neural networks be used for sentence representation instead of recurrent NN ?

I know that we can generate sentence representation using Bag of words (taking the summation of the word vectors) or using recurrent neural networks (LSTM or GRU). I am new to recursive NN and NLP. Is ...
0
votes
1answer
73 views

Dimensionality reduction with least distance distortion

Question: Could I find a dimensionality reduction algorithm without or with minimal distance (cosine) distortion? Background: I would like to visualize in 2D a sample of news texts for which I also ...
0
votes
0answers
9 views

Word Embeddings output from same algorithm has same vector representation?

I would like to know if output of word embeddings from same algorithm has same feature orientation or not. For example, if $V_1 = [1.924,2.323,3.456]$ and $V_2 = [1.987,3.212,7.676]$ are outputs of ...
1
vote
0answers
99 views

Could we drop the hidden layer in a skip-gram word2vec and train only a square weight matrix?

After pondering on the (skip-gram) word2vec algorithm and the fact that its single hidden layer is linearly activated, I am not 100% sure that I understand the significance of everything that is ...
0
votes
0answers
163 views

What is the optimal sampling rate / window size for Word2Vec Continuous Bag of Words?

What is the greatest number of embeddings you can average for the Word2Vec Cbow algorithm before measures of quality start dropping? For skip-gram I've seen window sizes up to 20 work, but I imagine ...
0
votes
0answers
34 views

why is negative sampling more often used in skip2gram rather than cbow

The first time I know negative sampling was from Skip2gram and the goal of negative sampling is to help the model not only learn what is correct but also what is wrong. But why negative sampling is ...
2
votes
1answer
413 views

Word2Vec and PyTorch - am I approaching this correctly?

My understanding of Word2Vec is that the library allows for generation of an array of numbers that approximates the meaning of a word relative to others in a sentence. My use of Word2Vec e.g. ...
0
votes
0answers
32 views

Word embedding as input or raw text?

I'm trying to implement a neural network for text recognition and I'm a little bit confused about text inputs. The goal of the network is to classify a comment, toy example: ...
0
votes
1answer
142 views

Classification using n-grams

I have $10000$ samples of 6-lettered strings of the following type Left                  Right  &...
0
votes
2answers
198 views

Anomaly detection in Text Classification

I have built a text classifier using OneClassSVM. I have the training set which corresponds to only one label i.e("Yes") and I don't have the other("NO") label data. My task is to build a classifier ...
2
votes
1answer
209 views

Can I use word2vec vectors as input features to NMF or LDA?

I'm trying to do some topic modelling on my corpus and I want to use Word2Vec vectors as an input to my NMF and LDA models. How do I do this? Is it even possible?
0
votes
1answer
472 views

Is skip-gram model in word2vec an expanded version of N-Gram model? skip-gram vs. skip-grams?

The skip-gram model of word2vec uses a shallow neural network to learn the word embedding with (input-word, context-word) data. When I read the tutorials for the skip-gram model there was not any ...
1
vote
0answers
25 views

Machine Learning: mathematical verification of this text-to-image cross entropy loss function?

I'm implementing a research paper on GANs and have come across this rather convoluted text-image loss function which has these main components: $$P(D_i | Q_i) = \frac{\exp({\gamma_3 R(Q_i, D_i)})}{\...
0
votes
1answer
28 views

Unsupervised answering for a predefined set of questions

I am working on a project to read up a text segment and find answers to a specific set of questions, in order to do some information extraction. I have a set of text corpus (each of about 3000 words),...
1
vote
0answers
121 views

Why not detect convergence in word2vec (skip-gram and cbow)?

In the word2vec software, as well as the implementation in gensim, training is done for a given number of epochs, and the learning rate (alpha) is decreased every 10000 words till a minimal alpha. ...
1
vote
1answer
130 views

use pre-trained word2vec to create the wordvector

I'm quite new to machine learning and Nlp. I want to do my project using word2vec. let say I hava word vector [ [drawing,painting], [reading,game,assembly]] this represent the person1 and person2'...
0
votes
0answers
144 views

How do I add a missing word to a pretrained embedding?

I have a pretrained word embedding and want to add missing words to it. How exactly should I do that? I think to just randomly initialize the vector is not a good idea. I heard something about ...
0
votes
0answers
33 views

Clustering industry field

I have a big company dataset with an industry variable that looks like this: ...
0
votes
1answer
24 views

Extract/detect IDs (like flight booking ids) from text

I'm looking to extract ids from the body of an email. The ids are similar to flight booking IDs. For example, in an email, I would like to obtain the booking reference (something like MNFF3RGC or MNF-...
0
votes
0answers
25 views

Using Lift to filter Transition Matrix for Word2Vec

Can we equate likelihood $P( A | B )$, to non-self cyclical, i.e.: $P ( A | A ) = 0$ transition matrix in page rank? If yes, does it make sense to ignore pairs with $\text{lift} \le 1$, where $$\text{...
0
votes
0answers
133 views

How can I do word/ char embeddings?

I'm working on a paper where it says that words and then characters ( each in a separate phase ) will be represented by d-dimensional randomly initialized embedding. " we build word and character ...
0
votes
2answers
288 views

Performing Word Embeddings with domain-specific data

I am new to word-embeddings and have only worked with older approaches like bag of words/tf-idf. Unlike td-idf or bag of words, I have to first train a model to perform the embeddings. If working ...
1
vote
0answers
50 views

Is there a pre-trained word embedding for english song lyrics?

I'm working on a project where the dataset is English songs. So I need word embeddings which are trained on English songs. If none exist, Could you please suggest one that matches for this use case?
1
vote
1answer
603 views

What is the meaning of the average value of all word vectors in the sentence?

Today I saw a sentiment analysis article here. There is one piece hard to understand: Next we have to build word vectors for input text in order to average the value of all word vectors in the ...
1
vote
0answers
50 views

Great ways to identify adult content in text

What are some good ways to identify adult content in text. It is definitely a text classification problem, but how do we handle words that are spelt like @$$.
0
votes
1answer
82 views

Query about Word2Vec

Two basic questions about word2vec. While training a skip-gram word2vec model, is the training data 1-to-1 or 1-to-many, i.e., say we have a sentence "the quick brown fox jumps over the lazy dog .." ...
1
vote
1answer
221 views

Does hierarchical softmax of skip gram and CBOW only update output vectors on the path from the root to the actual output word?

After reading word2vec Parameter Learning Explained by Xin Rong, I understand that in the hierarchical softmax model, there is no output vector representation for words, instead, each of the $V-1$ ...
2
votes
0answers
68 views

Word Association in text mining [closed]

I want to extract the information from text on the basis of association like. "Shahrukh Khan is the Famous actor of Bollywood, his wife name is Gauri khan. He is 52 years old. Sample output like as <...
0
votes
0answers
36 views

Training Word Vectors on whole corpus

I am training word2vec model on my corpus and a friend of mine asked me if it is right to train the word2vec model on the whole corpus? Because when creating word embeddings I am using the whole ...