Questions tagged [word2vec]

Word2vec is a neural network that represents words as vectors in a high dimensional space.

Filter by
Sorted by
Tagged with
1
vote
1answer
12 views

Why CBOW model is called “continuous”?

The question is pretty clear from the Title itself, why the Continuous Bag of Words (CBOW) model is called continuous. I also don't know what exactly "distributed representation" of word vectors ...
1
vote
0answers
11 views

Detecting anomalies in system logs

Chances are this may be closed off as too broad, but I'll try to be as specific as I can. I am currently working with API logs with categorical features separated by 1ms intervals, an example: ...
1
vote
0answers
22 views

How does using the same vector for the center word and for the context word impact the performance of word vectors in word2vec?

By default, word2vec uses 2 vectors for each word: one for the center word and one for the context word: $\color{steelblue}{\large \text{Word2vec: objective function}}$ $\color{darkred}{\...
0
votes
1answer
76 views

Multi class classification using Naive Bayes

I have components basically divided into two main categories. AWS and Azure. For eg: ...
0
votes
0answers
15 views

What Keras models would be best for output of lists of word vectors?

Imagine a regression model that is to be trained on data consisting of questions and answers expressed in text. The questions and answers are converted to lists of word vectors using some good word ...
0
votes
0answers
17 views

How the values in word2vec embedding are created for each word

I was going through word2vec materials from Andrew Ng's course and below is what i understood. -> Step1 A matrix of shape embedding_size*number_of_unique_words is created and populated with random ...
0
votes
0answers
15 views

Word2Vec: Reliance on only one out of two weight matrices

Word2Vec is commonly used to identify words similar to a given input term. My understanding is that Word2Vec is particularly efficient in this task because it uses the word embeddings learned during ...
2
votes
1answer
45 views

Confusion on how skip gram implementation is formulated

I'm using this source to understand the skip gram model. Let's say the context size is $4$ ($2$ context words on each side of the target word). This image illustrates how training examples are ...
0
votes
0answers
37 views

Unable to learn weights of a Word2Vec model [duplicate]

I was going to implement a word embedding model - namely Word2Vec - by following this TensorFlow tutorial and adapting the code a little bit. Unfortunately, though, my model won't learn anything. I've ...
1
vote
0answers
17 views

How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

This is related to my earlier question, which I'm trying to break down into parts (this being the first) since it seemed too large. I'm reading notes on word vectors here. Specifically, I'm referring ...
1
vote
0answers
15 views

Understanding how continuous bag of words method learns embedded representations

I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding: If we have a vocabulary $V$, the naive way to ...
0
votes
1answer
56 views

How to create word vectors from short sentences having mixed language (English and Hinglish)?

How do I create word vectors from a corpus where sentences are very short. e.g if the corpus contains messages from users - 'good morning', 'hello!', 'No, I can't.', 'Where?' etc. One way to resolve ...
0
votes
0answers
13 views

Doc2Vec models comparing $t$ and $t-1$

I have data for 10 years. Each year contains a set of documents $D$ in $t$. Each document consists of text and corresponds to a unique ID which does not change over time $T$. So ID $7814$ in year $3$ ...
0
votes
0answers
49 views

node2vec: Intuition behind BFS resulting in embeddings that capture structural equivalence

In the node2vec paper1 it is mentioned that when using BFS to embed nodes, the results correspond to structural equivalence (i.e. nodes that are "bridge nodes" would get embedded close together) ...
1
vote
0answers
16 views

In Skip-Gram model, what is the intuition of taking average of input word matrix and output word matrix?

In the skip-gram model, we have input word matrix $W$, and output word matrix $W'$. The final word embedding we got, is the average of $W$ and $W'$. what is the intuition of taking average of $W$ and $...
1
vote
0answers
65 views

How can I define a accuracy measure for word2vec predictions

I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
1
vote
1answer
65 views

Is negative sampling only used for computational reasons?

Is negative sampling only used for computational reasons in word2vec and other embedding algorithms? Or are there other benefits?
0
votes
1answer
40 views

What exactly is Word2Vec in the context of CBoW and Skip-gram?

I was reading the original paper for Word2Vec (Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013)) and got confused regarding Word2Vec, CBoW, and Skip-...
0
votes
1answer
20 views

How are word vector representations derived in Skip-gram?

I was reading the paper Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013 NIPS) and came across a part that I cannot quite understand. Specifically, ...
1
vote
1answer
38 views

How is addition and subtraction determined when using word embeddings like Word2Vec?

This question is particularly in the context of the word embedding algorithm Word2Vec. I've noticed that many examples that are given in the original paper and other blog posts say things like ...
0
votes
1answer
14 views

Word coocurrence in word2vec

Based on the vector values of two words in word2vec, could we judge whether they co-occur and frequencies of coocurrence?
1
vote
1answer
108 views

Word2vec Skip-Gram - Overfitting

i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm: So my model creates each run nearly ...
3
votes
1answer
26 views

Is it possible to take a pertained word embedding, trained on a general vocabulary and make it domain specific?

Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus ...
0
votes
0answers
15 views

In the context of word vectors what is bias?

I'm attempting to understand how bias can be measured using word embeddings. Reading the article https://towardsdatascience.com/gender-bias-word-embeddings-76d9806a0e17 What is the bias being ...
0
votes
0answers
30 views

Improve the accuracy of semantic text matching

I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning. I tried several approaches: using ...
0
votes
0answers
88 views

Word2Vec: Why use input embeddings instead of output embeddings in the skip-gram model?

When using the skip-gram model to learn word embeddings, the algorithm returns two types of embeddings for each word, an input embedding and an output embedding. Usually the input embedding is used ...
0
votes
1answer
183 views

Efficiently normalize word embeddings

I'm using glove word embedding and would like to [-1,1] normalize it using python. The data is in the format of a dict with the word as key and a ...
2
votes
0answers
76 views

Can you use VAEs to produce deep word embeddings?

There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
0
votes
0answers
57 views

Does skipgram(word2vec) decreases the euclidean distance or increases cosine similarity between similar words?

The skip-gram model tends to predict the surrounding words or in other words, it tries to maximize the co-occurrence in the output of the Network. According to my knowledge, this makes a similar word ...
0
votes
0answers
33 views

Visualising sentence vectors by averaging word vectors

I have $82114$ sentences for which I have found the vector representation by summing over individual word vectors(using Word2Vec). Now I have a vector representation for each sentence in my dataset. ...
0
votes
0answers
55 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
1
vote
0answers
51 views

Finding similarity dissimilarity between different groups of vectors

Suppose I need to combine or group together set of vectors in one area and another group in other areas, however I need to place these groups in a plot so they are scattered in the screen per ...
0
votes
0answers
29 views

Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
0
votes
0answers
20 views

contextual embedding algorithm including continuous variables

Is there any work that allows contextual embedding of events that allows not only for categorical information but also some magnitude information (as opposed to word2vec, GloVe etc) I have a series ...
0
votes
1answer
152 views

Understanding word2vec backpropagation

I'm watching the following video on word2vec from University of Waterloo: https://www.youtube.com/watch?v=GMCwS7tS5ZM&t=962s The update function for word my word embedding vector is: $v'_w = v_w ...
0
votes
0answers
58 views

word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
0
votes
0answers
72 views

Clustering positive and negative qualifiers with word2vec

I am looking to find whether a potential qualifier is positive, negative, or unknown. Example positive qualifiers are: increase, positive, raise. Example negative qualifiers are: decrease, negative, ...
0
votes
1answer
287 views

Should I use pretrained word2vec or train word2vec on my own dataset? [closed]

I am trying to perfrom fake news detection using machine learning naive bayes classifier. So far I have used BOW and TFIDF as my feature vectors. From research I have found that word embeddings plays ...
0
votes
0answers
47 views

Learning similarity of representations

I am interested in a framework for mapping together input representations based on some common context. I have looked into word2vec, which does more or less what I want, but I want to know if anyone ...
1
vote
1answer
213 views

Output from Word2Vec

I am working on sentiment analysis. I am using Word2Vec method. I don't understand the output from this code line. ...
0
votes
1answer
35 views

What is the motivation to train one's own word embedding model?

I've been using a few big word embedding models like word2vec & FastText, and they work very well on most problems. I am now adressing a new kind of data, on which they perform quite poorly, and I ...
0
votes
0answers
150 views

Can an embedding layer be replaced by a fully connected layer?

Due to architecture choices and organization of code, I have a file called data.py that processes texts and returns two vectors : X and Y which are the vectorized text and the corresponding label. ...
0
votes
1answer
147 views

Which layer is saved by CBOW? [duplicate]

The word2vec model saves its layer weights as embeddings. But do CBOW and skipgram both store the input layer weights? I know they learn different embeddings for the words in the context and for the ...
1
vote
0answers
10 views

Why does the “window-based” model fail to take advantage of the repetition?

In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says "Unlike the matrix factorization methods, the shallow window-based methods suffer from the disadvantage that they do not ...
1
vote
1answer
66 views

How does Word2Vec ensure that antonyms will be far apart in the vector space

Broadly speaking the training of word2vec is a process in which words that are often in the same context are clustered together in the vector space. We start by randomly shuffling the words on the ...
0
votes
1answer
33 views

Negative values in word vectorizations

I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
1
vote
0answers
68 views

How many word2vec pretrained models are available?

In my experiments with pre-trained word2vec models for NLP tasks, I have so far come across two models - one trained on Google News dataset and another which has been trained on Wikipedia text corpus. ...
1
vote
1answer
139 views

Dimensionality reduction with least distance distortion

Question: Could I find a dimensionality reduction algorithm without or with minimal distance (cosine) distortion? Background: I would like to visualize in 2D a sample of news texts for which I also ...
1
vote
0answers
158 views

Could we drop the hidden layer in a skip-gram word2vec and train only a square weight matrix?

After pondering on the (skip-gram) word2vec algorithm and the fact that its single hidden layer is linearly activated, I am not 100% sure that I understand the significance of everything that is ...
0
votes
0answers
318 views

What is the optimal sampling rate / window size for Word2Vec Continuous Bag of Words?

What is the greatest number of embeddings you can average for the Word2Vec Cbow algorithm before measures of quality start dropping? For skip-gram I've seen window sizes up to 20 work, but I imagine ...