Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Doubt on formulating cost function for GloVe

I'm reading the notes here and have a doubt on page 2 ("Least squares objective" section). The probability of a word $j$ occurring in the context of word $i$ is $$Q_{ij}=\frac{\exp(u_j^Tv_i)}{\sum_{w=...
1
vote
1answer
32 views

Text classification with small dataset for a specialized domain

I have a multiclass text classification problem where I have very few documents for each class. The classes are imbalanced but I want to be able to predict the class when I have at least 200 - 300 ...
0
votes
1answer
10 views

Is learning label embedding by factorizing label co-occurrence matrix unsupervised learning?

I was working on creating embeddings for medical concepts. These terms/phrases are used for annotating biomedical documents. Now usually the method of creating a co-occurrence matrix and then ...
0
votes
0answers
13 views

Should out-of-vocabulary words be skipped in the inference phase?

In the NLP preprocessing before a word embedding layer, the words or tokens not in the vocabulary are replaced with a out-of-vocab (OOV) token, in the training set. In the inference phase, I'm ...
0
votes
1answer
35 views

What is first order difference in trend analysis?

I am following this paper: Measuring, Predicting and Visualizing Short-Term Change in Word Representation and Usage in VKontakte Social Network where in Differencing Statistics section they describe ...
0
votes
0answers
37 views

Unable to learn weights of a Word2Vec model [duplicate]

I was going to implement a word embedding model - namely Word2Vec - by following this TensorFlow tutorial and adapting the code a little bit. Unfortunately, though, my model won't learn anything. I've ...
1
vote
0answers
14 views

How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

This is related to my earlier question, which I'm trying to break down into parts (this being the first) since it seemed too large. I'm reading notes on word vectors here. Specifically, I'm referring ...
3
votes
2answers
48 views

Inverse word embedding: vector to word

I'm building a generative text model, and the output of one of the final layers is a word embedding (vector) of the generated word. I'm left with the task of converting this vector back to the actual ...
1
vote
0answers
10 views

How to learn embeddings from lists of data?

My objective is to learn an embedding for translate sentences from one language to another. The problem is that my data looks like this: ...
1
vote
0answers
14 views

Understanding how continuous bag of words method learns embedded representations

I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding: If we have a vocabulary $V$, the naive way to ...
0
votes
1answer
22 views

How to create word vectors from short sentences having mixed language (English and Hinglish)?

How do I create word vectors from a corpus where sentences are very short. e.g if the corpus contains messages from users - 'good morning', 'hello!', 'No, I can't.', 'Where?' etc. One way to resolve ...
0
votes
0answers
4 views

Why does a non-decreasing weighting function ensure that rare word co-occurrences are not overweighted?

I was reading the original paper for the GloVe model and had a question. One page 4 of the paper under section 3: The GloVe Model, there is a portion that details properties that the authors' ...
1
vote
0answers
21 views

How can I define a accuracy measure for word2vec predictions

I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
1
vote
1answer
41 views

Is negative sampling only used for computational reasons?

Is negative sampling only used for computational reasons in word2vec and other embedding algorithms? Or are there other benefits?
0
votes
1answer
13 views

How are word vector representations derived in Skip-gram?

I was reading the paper Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013 NIPS) and came across a part that I cannot quite understand. Specifically, ...
1
vote
1answer
27 views

How is addition and subtraction determined when using word embeddings like Word2Vec?

This question is particularly in the context of the word embedding algorithm Word2Vec. I've noticed that many examples that are given in the original paper and other blog posts say things like ...
0
votes
0answers
7 views

Training a Transformer-LM with small data from scratch

I have a small dataset on a very specific domain. Does it makes sense to train a transformer language model from scratch on it? Are self-attention models harder to train (more data hungry) than older ...
0
votes
1answer
33 views

Word2vec Skip-Gram - Overfitting

i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm: So my model creates each run nearly ...
0
votes
1answer
36 views

Evaluating Word Embeddings: Expected Cosine Distance

One way to evaluate the quality of word embeddings is with tuples $(a, b, c, d)$ of words of analog relations of $a$ to $b$ and $c$ to $d$, such as ...
0
votes
1answer
21 views

Is Elmo equivalent to Fasttext+Bi-directional GRU?

From what I have read, Elmo uses bi-directional LSTM layers to give contextual embeddings for words in a sentence. So if I use a bi-directional LSTM/GRU layer over Fasttext representations of words, ...
3
votes
1answer
23 views

Is it possible to take a pertained word embedding, trained on a general vocabulary and make it domain specific?

Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus ...
0
votes
0answers
8 views

In the context of word vectors what is bias?

I'm attempting to understand how bias can be measured using word embeddings. Reading the article https://towardsdatascience.com/gender-bias-word-embeddings-76d9806a0e17 What is the bias being ...
1
vote
1answer
249 views

Is the Keras Embedding layer dependent on the target label?

I learned how to 'use' the Keras Embedding layer, but I am not able to find any more specific information about the actual behavior and training process of this layer. For now, I understand that the ...
2
votes
1answer
34 views

Balanced datasets are almost all predicted negative

Problem I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to ...
0
votes
0answers
29 views

Improve the accuracy of semantic text matching

I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning. I tried several approaches: using ...
1
vote
1answer
67 views

What is the intuition behind the positional cosine encoding in the transformer network?

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence. ...
0
votes
0answers
8 views

Creating a zero element in embedding space

I have some variable length input vectors for my own use case of a 'stylistic transfer'-esque process, and I am wondering if anyone knows of a way to engineer an input that maps to a 0 element in ...
0
votes
0answers
66 views

Word2Vec: Why use input embeddings instead of output embeddings in the skip-gram model?

When using the skip-gram model to learn word embeddings, the algorithm returns two types of embeddings for each word, an input embedding and an output embedding. Usually the input embedding is used ...
0
votes
1answer
92 views

Efficiently normalize word embeddings

I'm using glove word embedding and would like to [-1,1] normalize it using python. The data is in the format of a dict with the word as key and a ...
2
votes
0answers
59 views

Can you use VAEs to produce deep word embeddings?

There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
0
votes
1answer
34 views

Attention weights for identifying influence

This question concerns the application of self-attention weights for identifying influence of words in sentences. For instance, we are performing a classification task on a set of sentences (e.g. ...
0
votes
0answers
33 views

Does skipgram(word2vec) decreases the euclidean distance or increases cosine similarity between similar words?

The skip-gram model tends to predict the surrounding words or in other words, it tries to maximize the co-occurrence in the output of the Network. According to my knowledge, this makes a similar word ...
0
votes
0answers
26 views

Visualising sentence vectors by averaging word vectors

I have $82114$ sentences for which I have found the vector representation by summing over individual word vectors(using Word2Vec). Now I have a vector representation for each sentence in my dataset. ...
2
votes
4answers
411 views

Text Embeddings on a Small Dataset

I am trying to solve a binary text classification problem of academic text in a niche domain (Generative vs Cognitive Linguistics). My target text data consists of near 400 paper abstracts with less ...
0
votes
0answers
26 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
0answers
16 views

Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
0
votes
0answers
53 views

word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
0
votes
1answer
15 views

preparing free text column for regression

I have a column X which contains occupation/profession as an independent variable as free text, which is very much correlated with a continuous dependent variable. What techniques do you usually use ...
0
votes
1answer
188 views

Should I use pretrained word2vec or train word2vec on my own dataset? [closed]

I am trying to perfrom fake news detection using machine learning naive bayes classifier. So far I have used BOW and TFIDF as my feature vectors. From research I have found that word embeddings plays ...
0
votes
0answers
71 views

how to prepare text data for LSTM autoencoder

My main goal is to come up with some topics using LSTM autoencoder. I want to use 20 news_group data set. after reading lots of material and looking at some GitHub project, I am still not clear how ...
0
votes
1answer
24 views

What is the motivation to train one's own word embedding model?

I've been using a few big word embedding models like word2vec & FastText, and they work very well on most problems. I am now adressing a new kind of data, on which they perform quite poorly, and I ...
2
votes
1answer
52 views

Word Embedding for Sentiment Analysis

I am working on sentiment analysis of text. I am using keras word embedding. If my embedding has a vocabulary of 50 and an input length of 4 and I choose an embedding space of 8 dimensions, how will ...
0
votes
0answers
134 views

Can an embedding layer be replaced by a fully connected layer?

Due to architecture choices and organization of code, I have a file called data.py that processes texts and returns two vectors : X and Y which are the vectorized text and the corresponding label. ...
0
votes
1answer
89 views

Which layer is saved by CBOW?

The word2vec model saves its layer weights as embeddings. But do CBOW and skipgram both store the input layer weights? I know they learn different embeddings for the words in the context and for the ...
0
votes
1answer
73 views

siginificance testing of cosine similarity between word pairs from word embedding

I want to perform a significance test between two pairs of word embedding generated using Skip-gram model, as to how significant their cosine score is from mean similarity score. In order to do that I ...
1
vote
0answers
9 views

Why does the “window-based” model fail to take advantage of the repetition?

In Glove paper https://nlp.stanford.edu/pubs/glove.pdf, the author says "Unlike the matrix factorization methods, the shallow window-based methods suffer from the disadvantage that they do not ...
0
votes
0answers
57 views

Getting the document-per-topic loading using TextmineR package by passing term co-occurrence matrix

I am using TextmineR package to find the most similar documents to given document list. I used the following code to generate the tcm not dtm ...
0
votes
1answer
20 views

Modeling words in a language based on their characters

I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
2
votes
1answer
232 views

Facebook's infersent intuition

When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with: (u, v) the ...
2
votes
1answer
346 views

Clarification: text2vec, embeddings, doc2vec

I am trying to grasp the concept of word / document embeddings; I am using R as coding language, and I try to understand the text2vec package. Are the following statements about text2vec correct? ...