Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Why does a non-decreasing weighting function ensure that rare word co-occurrences are not overweighted?

I was reading the original paper for the GloVe model and had a question. One page 4 of the paper under section 3: The GloVe Model, there is a portion that details properties that the authors' ...
1
vote
0answers
15 views

How can I define a accuracy measure for word2vec predictions

I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
1
vote
1answer
26 views

Is negative sampling only used for computational reasons?

Is negative sampling only used for computational reasons in word2vec and other embedding algorithms? Or are there other benefits?
0
votes
1answer
11 views

How are word vector representations derived in Skip-gram?

I was reading the paper Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013 NIPS) and came across a part that I cannot quite understand. Specifically, ...
0
votes
1answer
21 views

How is addition and subtraction determined when using word embeddings like Word2Vec?

This question is particularly in the context of the word embedding algorithm Word2Vec. I've noticed that many examples that are given in the original paper and other blog posts say things like ...
0
votes
0answers
5 views

Training a Transformer-LM with small data from scratch

I have a small dataset on a very specific domain. Does it makes sense to train a transformer language model from scratch on it? Are self-attention models harder to train (more data hungry) than older ...
0
votes
0answers
14 views

Word2vec Skip-Gram - Overfitting

i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm: So my model creates each run nearly ...
0
votes
1answer
28 views

Evaluating Word Embeddings: Expected Cosine Distance

One way to evaluate the quality of word embeddings is with tuples $(a, b, c, d)$ of words of analog relations of $a$ to $b$ and $c$ to $d$, such as ...
0
votes
0answers
11 views

Is Elmo equivalent to Fasttext+Bi-directional GRU?

From what I have read, Elmo uses bi-directional LSTM layers to give contextual embeddings for words in a sentence. So if I use a bi-directional LSTM/GRU layer over Fasttext representations of words, ...
3
votes
1answer
18 views

Is it possible to take a pertained word embedding, trained on a general vocabulary and make it domain specific?

Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus ...
0
votes
0answers
8 views

In the context of word vectors what is bias?

I'm attempting to understand how bias can be measured using word embeddings. Reading the article https://towardsdatascience.com/gender-bias-word-embeddings-76d9806a0e17 What is the bias being ...
1
vote
1answer
130 views

Is the Keras Embedding layer dependent on the target label?

I learned how to 'use' the Keras Embedding layer, but I am not able to find any more specific information about the actual behavior and training process of this layer. For now, I understand that the ...
2
votes
1answer
33 views

Balanced datasets are almost all predicted negative

Problem I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to ...
0
votes
0answers
25 views

Improve the accuracy of semantic text matching

I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning. I tried several approaches: using ...
0
votes
1answer
26 views

What is the intuition behind the positional cosine encoding in the transformer network?

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence. ...
0
votes
0answers
8 views

Creating a zero element in embedding space

I have some variable length input vectors for my own use case of a 'stylistic transfer'-esque process, and I am wondering if anyone knows of a way to engineer an input that maps to a 0 element in ...
0
votes
0answers
45 views

Word2Vec: Why use input embeddings instead of output embeddings in the skip-gram model?

When using the skip-gram model to learn word embeddings, the algorithm returns two types of embeddings for each word, an input embedding and an output embedding. Usually the input embedding is used ...
0
votes
0answers
46 views

Efficiently normalize word embeddings

I'm using glove word embedding and would like to [-1,1] normalize it using python. The data is in the format of a dict with the word as key and a ...
1
vote
0answers
31 views

Can you use VAEs to produce deep word embeddings?

There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
0
votes
1answer
26 views

Attention weights for identifying influence

This question concerns the application of self-attention weights for identifying influence of words in sentences. For instance, we are performing a classification task on a set of sentences (e.g. ...
0
votes
0answers
24 views

Does skipgram(word2vec) decreases the euclidean distance or increases cosine similarity between similar words?

The skip-gram model tends to predict the surrounding words or in other words, it tries to maximize the co-occurrence in the output of the Network. According to my knowledge, this makes a similar word ...
0
votes
0answers
20 views

Visualising sentence vectors by averaging word vectors

I have $82114$ sentences for which I have found the vector representation by summing over individual word vectors(using Word2Vec). Now I have a vector representation for each sentence in my dataset. ...
2
votes
4answers
217 views

Text Embeddings on a Small Dataset

I am trying to solve a binary text classification problem of academic text in a niche domain (Generative vs Cognitive Linguistics). My target text data consists of near 400 paper abstracts with less ...
0
votes
0answers
17 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
0answers
15 views

Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
0
votes
0answers
44 views

word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
0
votes
1answer
14 views

preparing free text column for regression

I have a column X which contains occupation/profession as an independent variable as free text, which is very much correlated with a continuous dependent variable. What techniques do you usually use ...
0
votes
1answer
162 views

Should I use pretrained word2vec or train word2vec on my own dataset? [closed]

I am trying to perfrom fake news detection using machine learning naive bayes classifier. So far I have used BOW and TFIDF as my feature vectors. From research I have found that word embeddings plays ...
0
votes
0answers
56 views

how to prepare text data for LSTM autoencoder

My main goal is to come up with some topics using LSTM autoencoder. I want to use 20 news_group data set. after reading lots of material and looking at some GitHub project, I am still not clear how ...
0
votes
0answers
19 views

Word embeddings NLP

Studying about word embeddings I have a few questions because I have been confused. According some textbooks we have two categories of word embeddings the sparse models which based on frequency (word ...
0
votes
1answer
21 views

What is the motivation to train one's own word embedding model?

I've been using a few big word embedding models like word2vec & FastText, and they work very well on most problems. I am now adressing a new kind of data, on which they perform quite poorly, and I ...
2
votes
1answer
46 views

Word Embedding for Sentiment Analysis

I am working on sentiment analysis of text. I am using keras word embedding. If my embedding has a vocabulary of 50 and an input length of 4 and I choose an embedding space of 8 dimensions, how will ...
0
votes
0answers
108 views

Can an embedding layer be replaced by a fully connected layer?

Due to architecture choices and organization of code, I have a file called data.py that processes texts and returns two vectors : X and Y which are the vectorized text and the corresponding label. ...
0
votes
1answer
57 views

Which layer is saved by CBOW?

The word2vec model saves its layer weights as embeddings. But do CBOW and skipgram both store the input layer weights? I know they learn different embeddings for the words in the context and for the ...
0
votes
1answer
48 views

siginificance testing of cosine similarity between word pairs from word embedding

I want to perform a significance test between two pairs of word embedding generated using Skip-gram model, as to how significant their cosine score is from mean similarity score. In order to do that I ...
1
vote
0answers
7 views

Why does the “window-based” model fail to take advantage of the repetition?

In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says "Unlike the matrix factorization methods, the shallow window-based methods suffer from the disadvantage that they do not ...
0
votes
0answers
46 views

Getting the document-per-topic loading using TextmineR package by passing term co-occurrence matrix

I am using TextmineR package to find the most similar documents to given document list. I used the following code to generate the tcm not dtm ...
0
votes
1answer
19 views

Modeling words in a language based on their characters

I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
2
votes
1answer
194 views

Facebook's infersent intuition

When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with: (u, v) the ...
2
votes
1answer
294 views

Clarification: text2vec, embeddings, doc2vec

I am trying to grasp the concept of word / document embeddings; I am using R as coding language, and I try to understand the text2vec package. Are the following statements about text2vec correct? ...
0
votes
1answer
30 views

Negative values in word vectorizations

I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
1
vote
0answers
27 views

How many word2vec pretrained models are available?

In my experiments with pre-trained word2vec models for NLP tasks, I have so far come across two models - one trained on Google News dataset and another which has been trained on Wikipedia text corpus. ...
0
votes
0answers
87 views

Perplexity and accuracy

I am a little bit unsure of the results I am getting from my skip-gram model, as it says that I have a very high perplexity (>100k) but that the accuracy is also more than 50%. It seems a bit ...
0
votes
0answers
19 views

Linear projection learning for hypernymy with word embeddings

I want to train a model that, given the embedding of a hyponym, learns a projection (a transition matrix) to where it's hypernym should be in the embedding space. Ustalov et al. have proposed such a ...
2
votes
2answers
366 views

Using word embeddings in text classifier

I have a bunch of sentences that I want to do binary classification with SVM. My sentences have varying lengths form 4 to 34. If I use word embeddings such as word2vec or skip gram to convert my ...
1
vote
0answers
42 views

Deep Learning sentiment analysis model always predicts same class [closed]

I'd really appreciate your help as I'm not an expert in Deep Learning for sentiment analysis and I'm a bit lost. I'm using the Sentiment140 dataset: http://help.sentiment140.com/for-students First ...
0
votes
0answers
108 views

Can recursive neural networks be used for sentence representation instead of recurrent NN ?

I know that we can generate sentence representation using Bag of words (taking the summation of the word vectors) or using recurrent neural networks (LSTM or GRU). I am new to recursive NN and NLP. Is ...
0
votes
0answers
10 views

Word Embeddings output from same algorithm has same vector representation?

I would like to know if output of word embeddings from same algorithm has same feature orientation or not. For example, if $V_1 = [1.924,2.323,3.456]$ and $V_2 = [1.987,3.212,7.676]$ are outputs of ...
0
votes
0answers
7 views

How do I perform classification on instances that are sets of chronologically ordered texts?

I hope the question gives a bit of information regarding my goal. Let me first clarify a few things by giving some background. My goal is to perform classification on a set of texts at a time, ...
0
votes
0answers
137 views

keras embedding training optimization objective

I am aware of this and this existing questions, as well as this issue on github. Unless I am missing something though, all these fail to explain how the example in the keras docs makes sense: ...