Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

0
votes
0answers
7 views

Creating a zero element in embedding space

I have some variable length input vectors for my own use case of a 'stylistic transfer'-esque process, and I am wondering if anyone knows of a way to engineer an input that maps to a 0 element in ...
0
votes
0answers
11 views

Word2Vec: Why use input embeddings instead of output embeddings in the skip-gram model?

When using the skip-gram model to learn word embeddings, the algorithm returns two types of embeddings for each word, an input embedding and an output embedding. Usually the input embedding is used ...
0
votes
0answers
16 views

Efficiently normalize word embeddings

I'm using glove word embedding and would like to [-1,1] normalize it using python. The data is in the format of a dict with the word as key and a ...
1
vote
0answers
16 views

Can you use VAEs to produce deep word embeddings?

There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
0
votes
0answers
14 views

Attention weights for identifying influence

This question concerns the application of self-attention weights for identifying influence of words in sentences. For instance, we are performing a classification task on a set of sentences (e.g. ...
0
votes
0answers
12 views

Does skipgram(word2vec) decreases the euclidean distance or increases cosine similarity between similar words?

The skip-gram model tends to predict the surrounding words or in other words, it tries to maximize the co-occurrence in the output of the Network. According to my knowledge, this makes a similar word ...
0
votes
0answers
14 views

Visualising sentence vectors by averaging word vectors

I have $82114$ sentences for which I have found the vector representation by summing over individual word vectors(using Word2Vec). Now I have a vector representation for each sentence in my dataset. ...
2
votes
4answers
95 views

Text Embeddings on a Small Dataset

I am trying to solve a binary text classification problem of academic text in a niche domain (Generative vs Cognitive Linguistics). My target text data consists of near 400 paper abstracts with less ...
0
votes
0answers
5 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
0answers
8 views

Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
0
votes
0answers
27 views

word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
0
votes
1answer
13 views

preparing free text column for regression

I have a column X which contains occupation/profession as an independent variable as free text, which is very much correlated with a continuous dependent variable. What techniques do you usually use ...
0
votes
1answer
87 views

Should I use pretrained word2vec or train word2vec on my own dataset? [closed]

I am trying to perfrom fake news detection using machine learning naive bayes classifier. So far I have used BOW and TFIDF as my feature vectors. From research I have found that word embeddings plays ...
0
votes
0answers
31 views

how to prepare text data for LSTM autoencoder

My main goal is to come up with some topics using LSTM autoencoder. I want to use 20 news_group data set. after reading lots of material and looking at some GitHub project, I am still not clear how ...
0
votes
0answers
19 views

Word embeddings NLP

Studying about word embeddings I have a few questions because I have been confused. According some textbooks we have two categories of word embeddings the sparse models which based on frequency (word ...
0
votes
1answer
18 views

What is the motivation to train one's own word embedding model?

I've been using a few big word embedding models like word2vec & FastText, and they work very well on most problems. I am now adressing a new kind of data, on which they perform quite poorly, and I ...
2
votes
1answer
36 views

Word Embedding for Sentiment Analysis

I am working on sentiment analysis of text. I am using keras word embedding. If my embedding has a vocabulary of 50 and an input length of 4 and I choose an embedding space of 8 dimensions, how will ...
0
votes
0answers
70 views

Can an embedding layer be replaced by a fully connected layer?

Due to architecture choices and organization of code, I have a file called data.py that processes texts and returns two vectors : X and Y which are the vectorized text and the corresponding label. ...
0
votes
1answer
34 views

Which layer is saved by CBOW?

The word2vec model saves its layer weights as embeddings. But do CBOW and skipgram both store the input layer weights? I know they learn different embeddings for the words in the context and for the ...
0
votes
1answer
34 views

siginificance testing of cosine similarity between word pairs from word embedding

I want to perform a significance test between two pairs of word embedding generated using Skip-gram model, as to how significant their cosine score is from mean similarity score. In order to do that I ...
1
vote
0answers
6 views

Why does the “window-based” model fail to take advantage of the repetition?

In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says "Unlike the matrix factorization methods, the shallow window-based methods suffer from the disadvantage that they do not ...
0
votes
0answers
37 views

Getting the document-per-topic loading using TextmineR package by passing term co-occurrence matrix

I am using TextmineR package to find the most similar documents to given document list. I used the following code to generate the tcm not dtm ...
0
votes
1answer
18 views

Modeling words in a language based on their characters

I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
1
vote
1answer
107 views

Facebook's infersent intuition

When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with: (u, v) the ...
2
votes
1answer
216 views

Clarification: text2vec, embeddings, doc2vec

I am trying to grasp the concept of word / document embeddings; I am using R as coding language, and I try to understand the text2vec package. Are the following statements about text2vec correct? ...
0
votes
1answer
30 views

Negative values in word vectorizations

I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
1
vote
0answers
22 views

How many word2vec pretrained models are available?

In my experiments with pre-trained word2vec models for NLP tasks, I have so far come across two models - one trained on Google News dataset and another which has been trained on Wikipedia text corpus. ...
0
votes
0answers
67 views

Perplexity and accuracy

I am a little bit unsure of the results I am getting from my skip-gram model, as it says that I have a very high perplexity (>100k) but that the accuracy is also more than 50%. It seems a bit ...
0
votes
0answers
14 views

Linear projection learning for hypernymy with word embeddings

I want to train a model that, given the embedding of a hyponym, learns a projection (a transition matrix) to where it's hypernym should be in the embedding space. Ustalov et al. have proposed such a ...
2
votes
1answer
137 views

Using word embeddings in text classifier

I have a bunch of sentences that I want to do binary classification with SVM. My sentences have varying lengths form 4 to 34. If I use word embeddings such as word2vec or skip gram to convert my ...
1
vote
0answers
39 views

Deep Learning sentiment analysis model always predicts same class [closed]

I'd really appreciate your help as I'm not an expert in Deep Learning for sentiment analysis and I'm a bit lost. I'm using the Sentiment140 dataset: http://help.sentiment140.com/for-students First ...
0
votes
0answers
80 views

Can recursive neural networks be used for sentence representation instead of recurrent NN ?

I know that we can generate sentence representation using Bag of words (taking the summation of the word vectors) or using recurrent neural networks (LSTM or GRU). I am new to recursive NN and NLP. Is ...
0
votes
0answers
9 views

Word Embeddings output from same algorithm has same vector representation?

I would like to know if output of word embeddings from same algorithm has same feature orientation or not. For example, if $V_1 = [1.924,2.323,3.456]$ and $V_2 = [1.987,3.212,7.676]$ are outputs of ...
0
votes
0answers
7 views

How do I perform classification on instances that are sets of chronologically ordered texts?

I hope the question gives a bit of information regarding my goal. Let me first clarify a few things by giving some background. My goal is to perform classification on a set of texts at a time, ...
0
votes
0answers
126 views

keras embedding training optimization objective

I am aware of this and this existing questions, as well as this issue on github. Unless I am missing something though, all these fail to explain how the example in the keras docs makes sense: ...
0
votes
0answers
192 views

What is the optimal sampling rate / window size for Word2Vec Continuous Bag of Words?

What is the greatest number of embeddings you can average for the Word2Vec Cbow algorithm before measures of quality start dropping? For skip-gram I've seen window sizes up to 20 work, but I imagine ...
0
votes
0answers
9 views

Learning similarity metric from data

I want to measure the similarity of fastText vectors. Typically, for vector similarity, the cosine similarity is used. I would like to learn a notion of similarity based on the labels of the tokens ...
0
votes
1answer
19 views

Clarification about RNN/LSTM Sequence Models with Word Vector inputs

Say that we are trying to train a language model with an RNN/LSTM i.e. the inputs are words in a sentence and the outputs are the same words shifted by one such that for each input word the output is ...
0
votes
1answer
160 views

Classification using n-grams

I have $10000$ samples of 6-lettered strings of the following type Left                  Right  &...
1
vote
1answer
19 views

Group of word representations

For word representation baseline people use bag-of-words or word embedding. Here, I want to understand all approaches that can be used for word representations. For example: -Bag-of-words (tfidf, n-...
3
votes
2answers
78 views

Intuition behind word vector representations

How is it possible for a vector space to represent words so that it is coincident to our intuition of words? What does the orthogonality concept in such a space mean precisely? I think we can present ...
0
votes
1answer
17 views

How to have different source and target vocabularies?

I am a bit confused as to how to use different sets of source and target vocabularies in deep learning for NLP tasks. What are the implications of using separate source and target vocabularies (for ...
1
vote
0answers
34 views

Long repetitive output after changing vocabulary in seq2seq model

I trained a neural question generation model, which produces sensible questions for the vocabulary that they distributed with the paper. I wanted to run the model on a different set of word embeddings ...
0
votes
1answer
533 views

Is skip-gram model in word2vec an expanded version of N-Gram model? skip-gram vs. skip-grams?

The skip-gram model of word2vec uses a shallow neural network to learn the word embedding with (input-word, context-word) data. When I read the tutorials for the skip-gram model there was not any ...
0
votes
1answer
28 views

Unsupervised answering for a predefined set of questions

I am working on a project to read up a text segment and find answers to a specific set of questions, in order to do some information extraction. I have a set of text corpus (each of about 3000 words),...
1
vote
0answers
131 views

Why not detect convergence in word2vec (skip-gram and cbow)?

In the word2vec software, as well as the implementation in gensim, training is done for a given number of epochs, and the learning rate (alpha) is decreased every 10000 words till a minimal alpha. ...
4
votes
0answers
179 views

How are the internal representations in ELMo averaged?

I have been reading the paper "Deep contextualized word representations" (by Peters et al, 2018) to learn about the new embedding method called ELMo. In this paper, the authors train a charCNN + bi-...
1
vote
1answer
133 views

use pre-trained word2vec to create the wordvector

I'm quite new to machine learning and Nlp. I want to do my project using word2vec. let say I hava word vector [ [drawing,painting], [reading,game,assembly]] this represent the person1 and person2'...
0
votes
1answer
162 views

How do I add a missing word to a pretrained embedding?

I have a pretrained word embedding and want to add missing words to it. How exactly should I do that? I think to just randomly initialize the vector is not a good idea. I heard something about ...
0
votes
0answers
25 views

Using Lift to filter Transition Matrix for Word2Vec

Can we equate likelihood $P( A | B )$, to non-self cyclical, i.e.: $P ( A | A ) = 0$ transition matrix in page rank? If yes, does it make sense to ignore pairs with $\text{lift} \le 1$, where $$\text{...