Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

0
votes
0answers
4 views

Output from ELMO model

I am working on sentiment analysis of text. I started using ELMO word embeddings with tensor flow for it. I am using pretrained elmo model as given below elmo = hub.Module("https://tfhub.dev/...
0
votes
0answers
15 views

word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
0
votes
1answer
12 views

preparing free text column for regression

I have a column X which contains occupation/profession as an independent variable as free text, which is very much correlated with a continuous dependent variable. What techniques do you usually use ...
0
votes
1answer
27 views

Should I use pretrained word2vec or train word2vec on my own dataset?

I am trying to perfrom fake news detection using machine learning naive bayes classifier. So far I have used BOW and TFIDF as my feature vectors. From research I have found that word embeddings plays ...
0
votes
0answers
16 views

how to prepare text data for LSTM autoencoder

My main goal is to come up with some topics using LSTM autoencoder. I want to use 20 news_group data set. after reading lots of material and looking at some GitHub project, I am still not clear how ...
0
votes
0answers
18 views

Word embeddings NLP

Studying about word embeddings I have a few questions because I have been confused. According some textbooks we have two categories of word embeddings the sparse models which based on frequency (word ...
0
votes
1answer
12 views

What is the motivation to train one's own word embedding model?

I've been using a few big word embedding models like word2vec & FastText, and they work very well on most problems. I am now adressing a new kind of data, on which they perform quite poorly, and I ...
2
votes
1answer
30 views

Word Embedding for Sentiment Analysis

I am working on sentiment analysis of text. I am using keras word embedding. If my embedding has a vocabulary of 50 and an input length of 4 and I choose an embedding space of 8 dimensions, how will ...
0
votes
0answers
16 views

Can an embedding layer be replaced by a fully connected layer?

Due to architecture choices and organization of code, I have a file called data.py that processes texts and returns two vectors : X and Y which are the vectorized text and the corresponding label. ...
0
votes
1answer
15 views

Which layer is saved by CBOW?

The word2vec model saves its layer weights as embeddings. But do CBOW and skipgram both store the input layer weights? I know they learn different embeddings for the words in the context and for the ...
0
votes
1answer
19 views

siginificance testing of cosine similarity between word pairs from word embedding

I want to perform a significance test between two pairs of word embedding generated using Skip-gram model, as to how significant their cosine score is from mean similarity score. In order to do that I ...
1
vote
0answers
5 views

Why does the “window-based” model fail to take advantage of the repetition?

In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says "Unlike the matrix factorization methods, the shallow window-based methods suffer from the disadvantage that they do not ...
0
votes
0answers
21 views

Getting the document-per-topic loading using TextmineR package by passing term co-occurrence matrix

I am using TextmineR package to find the most similar documents to given document list. I used the following code to generate the tcm not dtm ...
0
votes
0answers
11 views

Modeling words in a language based on their characters

I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
1
vote
1answer
51 views

Facebook's infersent intuition

When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with: (u, v) the ...
1
vote
1answer
137 views

Clarification: text2vec, embeddings, doc2vec

I am trying to grasp the concept of word / document embeddings; I am using R as coding language, and I try to understand the text2vec package. Are the following statements about text2vec correct? ...
0
votes
1answer
28 views

Negative values in word vectorizations

I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
1
vote
0answers
20 views

How many word2vec pretrained models are available?

In my experiments with pre-trained word2vec models for NLP tasks, I have so far come across two models - one trained on Google News dataset and another which has been trained on Wikipedia text corpus. ...
0
votes
0answers
30 views

Perplexity and accuracy

I am a little bit unsure of the results I am getting from my skip-gram model, as it says that I have a very high perplexity (>100k) but that the accuracy is also more than 50%. It seems a bit ...
0
votes
0answers
12 views

Linear projection learning for hypernymy with word embeddings

I want to train a model that, given the embedding of a hyponym, learns a projection (a transition matrix) to where it's hypernym should be in the embedding space. Ustalov et al. have proposed such a ...
2
votes
1answer
34 views

Using word embeddings in text classifier

I have a bunch of sentences that I want to do binary classification with SVM. My sentences have varying lengths form 4 to 34. If I use word embeddings such as word2vec or skip gram to convert my ...
1
vote
0answers
32 views

Deep Learning sentiment analysis model always predicts same class [closed]

I'd really appreciate your help as I'm not an expert in Deep Learning for sentiment analysis and I'm a bit lost. I'm using the Sentiment140 dataset: http://help.sentiment140.com/for-students First ...
0
votes
0answers
59 views

Can recursive neural networks be used for sentence representation instead of recurrent NN ?

I know that we can generate sentence representation using Bag of words (taking the summation of the word vectors) or using recurrent neural networks (LSTM or GRU). I am new to recursive NN and NLP. Is ...
0
votes
0answers
9 views

Word Embeddings output from same algorithm has same vector representation?

I would like to know if output of word embeddings from same algorithm has same feature orientation or not. For example, if $V_1 = [1.924,2.323,3.456]$ and $V_2 = [1.987,3.212,7.676]$ are outputs of ...
0
votes
0answers
7 views

How do I perform classification on instances that are sets of chronologically ordered texts?

I hope the question gives a bit of information regarding my goal. Let me first clarify a few things by giving some background. My goal is to perform classification on a set of texts at a time, ...
0
votes
0answers
112 views

keras embedding training optimization objective

I am aware of this and this existing questions, as well as this issue on github. Unless I am missing something though, all these fail to explain how the example in the keras docs makes sense: ...
0
votes
0answers
117 views

What is the optimal sampling rate / window size for Word2Vec Continuous Bag of Words?

What is the greatest number of embeddings you can average for the Word2Vec Cbow algorithm before measures of quality start dropping? For skip-gram I've seen window sizes up to 20 work, but I imagine ...
0
votes
0answers
9 views

Learning similarity metric from data

I want to measure the similarity of fastText vectors. Typically, for vector similarity, the cosine similarity is used. I would like to learn a notion of similarity based on the labels of the tokens ...
0
votes
1answer
18 views

Clarification about RNN/LSTM Sequence Models with Word Vector inputs

Say that we are trying to train a language model with an RNN/LSTM i.e. the inputs are words in a sentence and the outputs are the same words shifted by one such that for each input word the output is ...
0
votes
1answer
129 views

Classification using n-grams

I have $10000$ samples of 6-lettered strings of the following type Left                  Right  &...
1
vote
1answer
19 views

Group of word representations

For word representation baseline people use bag-of-words or word embedding. Here, I want to understand all approaches that can be used for word representations. For example: -Bag-of-words (tfidf, n-...
3
votes
2answers
72 views

Intuition behind word vector representations

How is it possible for a vector space to represent words so that it is coincident to our intuition of words? What does the orthogonality concept in such a space mean precisely? I think we can present ...
0
votes
1answer
16 views

How to have different source and target vocabularies?

I am a bit confused as to how to use different sets of source and target vocabularies in deep learning for NLP tasks. What are the implications of using separate source and target vocabularies (for ...
1
vote
0answers
27 views

Long repetitive output after changing vocabulary in seq2seq model

I trained a neural question generation model, which produces sensible questions for the vocabulary that they distributed with the paper. I wanted to run the model on a different set of word embeddings ...
0
votes
1answer
372 views

Is skip-gram model in word2vec an expanded version of N-Gram model? skip-gram vs. skip-grams?

The skip-gram model of word2vec uses a shallow neural network to learn the word embedding with (input-word, context-word) data. When I read the tutorials for the skip-gram model there was not any ...
0
votes
1answer
26 views

Unsupervised answering for a predefined set of questions

I am working on a project to read up a text segment and find answers to a specific set of questions, in order to do some information extraction. I have a set of text corpus (each of about 3000 words),...
1
vote
0answers
104 views

Why not detect convergence in word2vec (skip-gram and cbow)?

In the word2vec software, as well as the implementation in gensim, training is done for a given number of epochs, and the learning rate (alpha) is decreased every 10000 words till a minimal alpha. ...
4
votes
0answers
174 views

How are the internal representations in ELMo averaged?

I have been reading the paper "Deep contextualized word representations" (by Peters et al, 2018) to learn about the new embedding method called ELMo. In this paper, the authors train a charCNN + bi-...
1
vote
1answer
123 views

use pre-trained word2vec to create the wordvector

I'm quite new to machine learning and Nlp. I want to do my project using word2vec. let say I hava word vector [ [drawing,painting], [reading,game,assembly]] this represent the person1 and person2'...
0
votes
0answers
131 views

How do I add a missing word to a pretrained embedding?

I have a pretrained word embedding and want to add missing words to it. How exactly should I do that? I think to just randomly initialize the vector is not a good idea. I heard something about ...
0
votes
0answers
24 views

Using Lift to filter Transition Matrix for Word2Vec

Can we equate likelihood $P( A | B )$, to non-self cyclical, i.e.: $P ( A | A ) = 0$ transition matrix in page rank? If yes, does it make sense to ignore pairs with $\text{lift} \le 1$, where $$\text{...
0
votes
0answers
117 views

How can I do word/ char embeddings?

I'm working on a paper where it says that words and then characters ( each in a separate phase ) will be represented by d-dimensional randomly initialized embedding. " we build word and character ...
0
votes
1answer
247 views

Performing Word Embeddings with domain-specific data

I am new to word-embeddings and have only worked with older approaches like bag of words/tf-idf. Unlike td-idf or bag of words, I have to first train a model to perform the embeddings. If working ...
0
votes
1answer
24 views

Specifying the neighborhood for Word-to-Abbreviation associations in corpus of document pairs

TL;DR: When implementing word co-occurence algorithms and word-embeddings, can you specify the area to be treated as the neighborhood for a word 'x' (i.e., treat a separate document as the ...
1
vote
1answer
76 views

How to compute context-independent token representations in a biLM?

I've been reading this paper on ELMo word representations. For context, here's my understanding of the standard bi-directional language model (biLM) thus far: Given a sequence of tokens $(t_{1}, ...
1
vote
0answers
44 views

Is there a pre-trained word embedding for english song lyrics?

I'm working on a project where the dataset is English songs. So I need word embeddings which are trained on English songs. If none exist, Could you please suggest one that matches for this use case?
0
votes
1answer
194 views

How to improve performance for LDA?

I am running LDA on health-related data. Specifically I have ~500 documents that contain interviews that last around 5-7 pages. Other than that, I cannot really go into the details of the data due to ...
0
votes
1answer
81 views

Query about Word2Vec

Two basic questions about word2vec. While training a skip-gram word2vec model, is the training data 1-to-1 or 1-to-many, i.e., say we have a sentence "the quick brown fox jumps over the lazy dog .." ...
0
votes
0answers
48 views

Loss function for partial/splitted softmax outputs on binary ground truth

I need to find a loss function for the scenario where each output is a vector instead of a scalar. And each of these vectors in one-hot-encoded. So I would like to use something like softmax loss on ...
1
vote
1answer
201 views

Does hierarchical softmax of skip gram and CBOW only update output vectors on the path from the root to the actual output word?

After reading word2vec Parameter Learning Explained by Xin Rong, I understand that in the hierarchical softmax model, there is no output vector representation for words, instead, each of the $V-1$ ...