Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
283 questions
0
votes
1
answer
543
views
Which layer is saved by CBOW? [duplicate]
The word2vec model saves its layer weights as embeddings.
But do CBOW and skipgram both store the input layer weights?
I know they learn different embeddings for the words in the context and for the ...
- 1
1
vote
0
answers
26
views
Why does the "window-based" model fail to take advantage of the repetition?
In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says
"Unlike the matrix factorization methods, the shallow window-based
methods suffer from the disadvantage that they do not ...
- 879
0
votes
1
answer
27
views
Modeling words in a language based on their characters
I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
- 13
5
votes
1
answer
1k
views
Facebook's infersent intuition
When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with:
(u, v) the ...
- 203
6
votes
1
answer
796
views
Clarification: text2vec, embeddings, doc2vec
I am trying to grasp the concept of word / document embeddings; I am using R as coding language, and I try to understand the text2vec package.
Are the following statements about text2vec correct?
...
- 621
0
votes
1
answer
991
views
Negative values in word vectorizations
I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
- 251
2
votes
1
answer
2k
views
How many word2vec pretrained models are available? [closed]
In my experiments with pre-trained word2vec models for NLP tasks, I have so far come across two models - one trained on Google News dataset and another which has been trained on Wikipedia text corpus.
...
2
votes
2
answers
2k
views
Using word embeddings in text classifier
I have a bunch of sentences that I want to do binary classification with SVM.
My sentences have varying lengths form 4 to 34. If I use word embeddings such as word2vec or skip gram to convert my ...
- 303
1
vote
0
answers
178
views
Deep Learning sentiment analysis model always predicts same class [closed]
I'd really appreciate your help as I'm not an expert in Deep Learning for sentiment analysis and I'm a bit lost.
I'm using the Sentiment140 dataset: http://help.sentiment140.com/for-students
First ...
- 41
0
votes
0
answers
311
views
keras embedding training optimization objective
I am aware of this and this existing questions, as well as this issue on github. Unless I am missing something though, all these fail to explain how the example in the keras docs makes sense:
...
- 201
1
vote
0
answers
1k
views
What is the optimal sampling rate / window size for Word2Vec Continuous Bag of Words?
What is the greatest number of embeddings you can average for the Word2Vec Cbow algorithm before measures of quality start dropping?
For skip-gram I've seen window sizes up to 20 work, but I imagine ...
- 1,229
1
vote
1
answer
101
views
Clarification about RNN/LSTM Sequence Models with Word Vector inputs
Say that we are trying to train a language model with an RNN/LSTM i.e. the inputs are words in a sentence and the outputs are the same words shifted by one such that for each input word the output is ...
- 1,895
1
vote
1
answer
22
views
Group of word representations
For word representation baseline people use bag-of-words or word embedding. Here, I want to understand all approaches that can be used for word representations. For example:
-Bag-of-words (tfidf, n-...
- 11
3
votes
2
answers
305
views
Intuition behind word vector representations
How is it possible for a vector space to represent words so that it is coincident to our intuition of words? What does the orthogonality concept in such a space mean precisely?
I think we can present ...
- 31
0
votes
1
answer
322
views
How to have different source and target vocabularies?
I am a bit confused as to how to use different sets of source and target vocabularies in deep learning for NLP tasks.
What are the implications of using separate source and target
vocabularies (for ...
1
vote
0
answers
133
views
Long repetitive output after changing vocabulary in seq2seq model
I trained a neural question generation model, which produces sensible questions for the vocabulary that they distributed with the paper. I wanted to run the model on a different set of word embeddings ...
4
votes
1
answer
4k
views
Is skip-gram model in word2vec an expanded version of N-Gram model? skip-gram vs. skip-grams?
The skip-gram model of word2vec uses a shallow neural network to learn the word embedding with (input-word, context-word) data. When I read the tutorials for the skip-gram model there was not any ...
- 2,485
0
votes
1
answer
30
views
Unsupervised answering for a predefined set of questions
I am working on a project to read up a text segment and find answers to a specific set of questions, in order to do some information extraction.
I have a set of text corpus (each of about 3000 words),...
- 121
2
votes
0
answers
312
views
Why not detect convergence in word2vec (skip-gram and cbow)?
In the word2vec software, as well as the implementation in gensim, training is done for a given number of epochs, and the learning rate (alpha) is decreased every 10000 words till a minimal alpha. ...
- 21
5
votes
1
answer
249
views
How are the internal representations in ELMo averaged?
I have been reading the paper "Deep contextualized word representations" (by Peters et al, 2018) to learn about the new embedding method called ELMo. In this paper, the authors train a charCNN + bi-...
2
votes
1
answer
800
views
use pre-trained word2vec to create the wordvector [closed]
I'm quite new to machine learning and Nlp.
I want to do my project using word2vec.
let say I hava word vector
[ [drawing,painting], [reading,game,assembly]]
this represent the person1 and person2'...
- 21
1
vote
1
answer
895
views
How do I add a missing word to a pretrained embedding?
I have a pretrained word embedding and want to add missing words to it. How exactly should I do that?
I think to just randomly initialize the vector is not a good idea.
I heard something about ...
- 253
2
votes
3
answers
3k
views
Performing Word Embeddings with domain-specific data
I am new to word-embeddings and have only worked with older approaches like bag of words/tf-idf. Unlike td-idf or bag of words, I have to first train a model to perform the embeddings.
If working ...
- 1,030
1
vote
1
answer
42
views
Specifying the neighborhood for Word-to-Abbreviation associations in corpus of document pairs
TL;DR: When implementing word co-occurence algorithms and word-embeddings, can you specify the area to be treated as the neighborhood for a word 'x' (i.e., treat a separate document as the ...
- 313
2
votes
1
answer
189
views
How to compute context-independent token representations in a biLM?
I've been reading this paper on ELMo word representations.
For context, here's my understanding of the standard bi-directional language model (biLM) thus far:
Given a sequence of tokens $(t_{1}, ...
1
vote
0
answers
125
views
Is there a pre-trained word embedding for english song lyrics?
I'm working on a project where the dataset is English songs. So I need word embeddings which are trained on English songs.
If none exist, Could you please suggest one that matches for this use case?
- 111
1
vote
1
answer
5k
views
How to improve performance for latent Dirichlet allocation (LDA)?
I am running LDA on health-related data. Specifically I have ~500 documents that contain interviews that last around 5-7 pages. Other than that, I cannot really go into the details of the data due to ...
- 1,030
1
vote
1
answer
287
views
Query about Word2Vec
Two basic questions about word2vec.
While training a skip-gram word2vec model, is the training data 1-to-1 or 1-to-many, i.e., say we have a sentence "the quick brown fox jumps over the lazy dog ...
- 858
3
votes
1
answer
1k
views
Does hierarchical softmax of skip gram and CBOW only update output vectors on the path from the root to the actual output word?
After reading word2vec Parameter Learning Explained by Xin Rong, I understand that in the hierarchical softmax model, there is no output vector representation for words, instead, each of the $V-1$ ...
- 620
1
vote
2
answers
1k
views
Losing Order with Word Embeddings?
I'm reading up on word embeddings and am a bit confused. It seems there are a couple approaches:
1) Use an unsupervised approach to generate word embeddings (basically predicting the probability of ...
- 533
1
vote
1
answer
47
views
A suitable corpus for training skip-though vectors
For training a variant of the notion of skip-though vectors, I need a long corpus of consecutive (related) sentences. The original skip-thought paper has used BookCorpus, but it is no longer available....
- 277
2
votes
1
answer
686
views
Algorithm for Debiasing Word Embeddings
I'm taking Andrew Ng's course on Sequence Models on Coursera, and he has a pretty fascinating discussion of how to "debias" word embeddings by removing the learned gender component (or race component, ...
- 1,106
2
votes
0
answers
336
views
Is there an extension of PCA for data embedded in hyperbolic spaces?
I'm working on a project where we are embedding data into an n-dimensional Poincare ball similar to this paper. However, we'd like to take the additional step of reducing this data to a 2-dimensional ...
- 21
1
vote
1
answer
315
views
What is the correct way to apply word embeddings to new data?
Consider the words "banana" and "split". Assume that a pre-trained word embedding (say, word2vec GoogleNews) has the vectors like so:
...
- 252
1
vote
1
answer
1k
views
NER at sentence level or document level?
Should NER models (LSTM or CRF) take input training data at sentence level or paragraph level?
Let's say we have this input text, and we would like to do Named Entity Extraction:
GOP Sen. Rand ...
- 63
2
votes
0
answers
116
views
can a word embedding encode two concepts in an order?
This question is from a homework in the Sequence Models course taught by Andrew Ng on Coursera (this is not related to a homework problem per se, just for general edification).
See the screenshot ...
- 1,106
7
votes
1
answer
3k
views
Word2Vec : Difference between the two Weight matrices
In Word2Vec algorithm, two weight matrices are learnt :
W : Input-hidden layer matrix
W': Hidden-output layer matrix
For reference, CBOW model architecture:
Why is W chosen to represent the word ...
- 73
1
vote
1
answer
377
views
Correctness of a skewed cosine similarity graph
I am currently implementing a word2vec model that uses the cosine similarity to determine the similarity between two vectors.
When plotting all the possible cosine similarities, I get the following ...
- 173
4
votes
1
answer
356
views
Skipgram - multiple formulations?
I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions.
1 - Taking a look at this blog post and Andrew Ng's Deep Learning Specialization, I understood ...
- 521
6
votes
0
answers
2k
views
Why word embeddings learned from word2vec are linearly correlated
I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino ...
- 1,176
1
vote
2
answers
3k
views
Learning image embeddings using VGG and Word2Vec
Background: In word2vec we pass in a one-hot encoding of our target word into a simple neural network which is trained to predict context words from a window around our target. We eventually take the ...
- 291
12
votes
3
answers
17k
views
What is difference between keras embedding layer and word2vec?
In other words, is there a paper that describes the method of keras embedding layer? Is there a comparison between these methods (and other methods like Glove etc.)?
- 1,017
3
votes
1
answer
830
views
How to extract vector representation from a comparison neural networks
From what I understand about embeddings in neural networks, the upper layers (fully-connected) from convolutional neural networks can serve as an image encoding (vector representation of an image), ...
- 377
2
votes
1
answer
666
views
Dimension reduction - word embeddings as inputs for a time series model (LSTM) [closed]
As a part of my Master Thesis I plan to use news headlines as an input feature for a time series model that predicts the daily trend of Bitcoin returns (1= positive return, 0= negative return). ...
- 61
3
votes
1
answer
1k
views
Which part of the hidden layer architecture do pretrained word embeddings come from?
I'm working on developing a better understanding of word embeddings, and am struggling a bit with understanding where pre-trained word embeddings come from. For instance, let's take Stanford's ...
- 251
1
vote
0
answers
287
views
Word embedding as features for classification
W.r.t text classification, a common approach is to combine (often sum or average) word embeddings and use the result vector as features.
Are there any reference document(s) that establish the ...
- 111
3
votes
1
answer
304
views
In Sequence to Sequence learning, how can large amounts of missing/special words in a sentence be compensated for?
I'm currently working on a Seq2Seq model for a chatbot and I'm converting every sentence to numerical vectors with word embeddings, i.e. GloVe.
My problem is that training doesn't progress; the ...
- 254
3
votes
0
answers
707
views
How to solve crossword puzzle by Machine Learning? [closed]
If we have enough crossword puzzles with answers in our database, is that possible to build a Machine Learning model to solve crossword puzzles?
I read the paper:
Learning to Understand Phrases by ...
- 439
27
votes
3
answers
22k
views
How the embedding layer is trained in Keras Embedding layer
How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)
Assume we do not use a pretrained embedding.
- 1,087
2
votes
1
answer
688
views
Why does the skipgram model takes more time to train compared to cbow?
I have been using gensim's word2vec implementation. Through my experiments, I learnt that Skipgram model takes 8x more time compared to CBOW on the same data, for the same number of dimensions and all ...
- 617