A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0
votes
1answer
19 views

Difference between non-contextual and contextual word embeddings

It is often stated that word2vec and GloVe are non-contextual embeddings while LSTM and Transformer-based (e.g. BERT) embeddings are contextual. The way I understand it, however, all word embeddings ...
0
votes
0answers
26 views

What to make of high R-squared and non-significant p-value of a linear model?

I am using doc2vec to produce $\mathbb{R}^{50}$ vector representations of short bits of text. I am then using those vectors in a linear model to predict a continuous outcome variable. The R^2 is .25 ...
1
vote
1answer
11 views

What is the target variable in the feed forward neural network within the Transformer model architecture?

In the paper 'Attention is all you need' the model architecture of The Transformer is described. Both in the encoder as well as in the decoder, there is a feed forward network. If I understant it ...
0
votes
0answers
6 views

Mapping patterns onto a set of very similar noisy sequences [closed]

I have a set of a sequence of patterns which need to be mapped against a set of sequences. The patterns have to occur in order. I.e. I want to fuzzily map an order of patterns (some regex) against a ...
0
votes
0answers
4 views

How to pass a Pandas column as a batch of sentences in a 1-D tensor of strings? [migrated]

I am struggling to pass a Pandas column (or numpy array) with size (2946, 1) to a Text embedding input layer in Keras with Tensorflow 2. The Pandas DataFrame object ...
0
votes
1answer
28 views

Train a RNN with unknown vocabulary size

I'm new to deep learning and i'm trying to code a Visual Question answering network. I studied and (i think) understood how RNN and LSTM work. From what i'he understood, i need to train my network ...
0
votes
0answers
9 views

I don't understand the task of finding the correlation in WordSimilarity-353

My issue is the following, I have created word embeddings using WordNet, and to test my embeddings to see how they stack up against the word similarities in WordSimilarity-353 I'm running a script ...
1
vote
0answers
22 views

How does using the same vector for the center word and for the context word impact the performance of word vectors in word2vec?

By default, word2vec uses 2 vectors for each word: one for the center word and one for the context word: $\color{steelblue}{\large \text{Word2vec: objective function}}$ $\color{darkred}{\...
1
vote
0answers
19 views

FastText or GloVe for code-mixed sentiment analysis?

I am currently working on a project for code-mixed sentiment analysis (English+Spanish). I've been using the GloVe Twitter word embeddings so far but I realized that even though this representation is ...
0
votes
0answers
19 views

What input for joint word and character embeddings

I'm implementing a neural network that classifies Tweets (positive/neutral/negative). I'm using GloVe Twitter word embeddings (200dim) but since I have a lot of OOV words I'd like to add to each ...
0
votes
0answers
5 views

Can Incremental/Online Learning be Implemented for Custom Word Embeddings

I'm currently working with a neural network (in Keras) that predicts classes from text using custom word embeddings. It's worked well until now, but has to be retrained frequently on new data. The ...
1
vote
1answer
46 views

How can this L(2,1) problem be reduced to the orthogonal procrustes problem?

NOTE: Don't take this too serious -- the question is actually due to my misreading $\|y_i - Wx_i\|^2$ as $\|y_i - Wx_i\|_2$, see the answer. Smith et al. in Offline bilingual word vectors, orthogonal ...
0
votes
1answer
66 views

batch-training LSTM with pretrained & out-of-vocabulary word embeddings in keras

My goal is to batch-train an RNN LSTM mode using Stochastic Gradient Descent to predict named entities from labeled text in keras. The input to my model are word-sized units. I chose to represent ...
1
vote
0answers
19 views

Contextual word embeddings to estimate likelihood of word given previous words in sentence?

I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
0
votes
0answers
15 views

What Keras models would be best for output of lists of word vectors?

Imagine a regression model that is to be trained on data consisting of questions and answers expressed in text. The questions and answers are converted to lists of word vectors using some good word ...
0
votes
0answers
17 views

How the values in word2vec embedding are created for each word

I was going through word2vec materials from Andrew Ng's course and below is what i understood. -> Step1 A matrix of shape embedding_size*number_of_unique_words is created and populated with random ...
0
votes
0answers
13 views

What does the dimension represent in the GloVe pre-trained word vectors?

I'm using GloVe pre-trained word vectors (glove.6b.50d.txt, glove.6b.300d.txt) to word embedding. I have a conceptual question: What is the difference between the mentioned files? On the other hand, ...
0
votes
1answer
32 views

Is skip-gram model of word embedding actually a multi-class task not a multi-label task, right?

So curious about this question, that I can't describe it in short. Please forgive me. Description: From multiclass and multilabel algorithms, we can get the definition of the multi-class and multi-...
1
vote
0answers
27 views

Deep learning model with both continuous and categorical features

I am working on a problem where the input contains both continuous and categorical features, and output is a numerical value. I convert each categorical feature into a fixed-dimension vector by one-...
0
votes
1answer
206 views

Understanding how word embedding with Fasttext works for my case

I'm looking for some guidance with Fasttext and NLP to help understand how the model proceed to calculate the vector of a sentence. Context: I'm using the fasttext method get_sentence_vector() to ...
4
votes
0answers
39 views

Workaround for word embeddings that do not “see” antonyms

Most word embeddings do not "see" antonyms. For instance, among many words they will place vectors for "dependent" and "independent" (as an example) quite close, - actually as close as with synonyms ...
0
votes
0answers
10 views

Doubt on formulating cost function for GloVe

I'm reading the notes here and have a doubt on page 2 ("Least squares objective" section). The probability of a word $j$ occurring in the context of word $i$ is $$Q_{ij}=\frac{\exp(u_j^Tv_i)}{\sum_{w=...
1
vote
2answers
114 views

Text classification with small dataset for a specialized domain

I have a multiclass text classification problem where I have very few documents for each class. The classes are imbalanced but I want to be able to predict the class when I have at least 200 - 300 ...
0
votes
1answer
15 views

Is learning label embedding by factorizing label co-occurrence matrix unsupervised learning?

I was working on creating embeddings for medical concepts. These terms/phrases are used for annotating biomedical documents. Now usually the method of creating a co-occurrence matrix and then ...
0
votes
0answers
15 views

Should out-of-vocabulary words be skipped in the inference phase?

In the NLP preprocessing before a word embedding layer, the words or tokens not in the vocabulary are replaced with a out-of-vocab (OOV) token, in the training set. In the inference phase, I'm ...
0
votes
1answer
54 views

What is first order difference in trend analysis?

I am following this paper: Measuring, Predicting and Visualizing Short-Term Change in Word Representation and Usage in VKontakte Social Network where in Differencing Statistics section they describe ...
0
votes
0answers
37 views

Unable to learn weights of a Word2Vec model [duplicate]

I was going to implement a word embedding model - namely Word2Vec - by following this TensorFlow tutorial and adapting the code a little bit. Unfortunately, though, my model won't learn anything. I've ...
1
vote
0answers
17 views

How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

This is related to my earlier question, which I'm trying to break down into parts (this being the first) since it seemed too large. I'm reading notes on word vectors here. Specifically, I'm referring ...
3
votes
2answers
139 views

Inverse word embedding: vector to word

I'm building a generative text model, and the output of one of the final layers is a word embedding (vector) of the generated word. I'm left with the task of converting this vector back to the actual ...
1
vote
0answers
14 views

How to learn embeddings from lists of data?

My objective is to learn an embedding for translate sentences from one language to another. The problem is that my data looks like this: ...
1
vote
0answers
15 views

Understanding how continuous bag of words method learns embedded representations

I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding: If we have a vocabulary $V$, the naive way to ...
0
votes
1answer
57 views

How to create word vectors from short sentences having mixed language (English and Hinglish)?

How do I create word vectors from a corpus where sentences are very short. e.g if the corpus contains messages from users - 'good morning', 'hello!', 'No, I can't.', 'Where?' etc. One way to resolve ...
0
votes
0answers
9 views

Why does a non-decreasing weighting function ensure that rare word co-occurrences are not overweighted?

I was reading the original paper for the GloVe model and had a question. One page 4 of the paper under section 3: The GloVe Model, there is a portion that details properties that the authors' ...
1
vote
0answers
67 views

How can I define a accuracy measure for word2vec predictions

I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
1
vote
1answer
65 views

Is negative sampling only used for computational reasons?

Is negative sampling only used for computational reasons in word2vec and other embedding algorithms? Or are there other benefits?
0
votes
1answer
20 views

How are word vector representations derived in Skip-gram?

I was reading the paper Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013 NIPS) and came across a part that I cannot quite understand. Specifically, ...
1
vote
1answer
39 views

How is addition and subtraction determined when using word embeddings like Word2Vec?

This question is particularly in the context of the word embedding algorithm Word2Vec. I've noticed that many examples that are given in the original paper and other blog posts say things like ...
0
votes
0answers
20 views

Training a Transformer-LM with small data from scratch

I have a small dataset on a very specific domain. Does it makes sense to train a transformer language model from scratch on it? Are self-attention models harder to train (more data hungry) than older ...
1
vote
1answer
112 views

Word2vec Skip-Gram - Overfitting

i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm: So my model creates each run nearly ...
0
votes
1answer
42 views

Evaluating Word Embeddings: Expected Cosine Distance

One way to evaluate the quality of word embeddings is with tuples $(a, b, c, d)$ of words of analog relations of $a$ to $b$ and $c$ to $d$, such as ...
0
votes
2answers
99 views

Is Elmo equivalent to Fasttext+Bi-directional GRU?

From what I have read, Elmo uses bi-directional LSTM layers to give contextual embeddings for words in a sentence. So if I use a bi-directional LSTM/GRU layer over Fasttext representations of words, ...
3
votes
1answer
26 views

Is it possible to take a pertained word embedding, trained on a general vocabulary and make it domain specific?

Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus ...
0
votes
0answers
15 views

In the context of word vectors what is bias?

I'm attempting to understand how bias can be measured using word embeddings. Reading the article https://towardsdatascience.com/gender-bias-word-embeddings-76d9806a0e17 What is the bias being ...
2
votes
1answer
470 views

Is the Keras Embedding layer dependent on the target label?

I learned how to 'use' the Keras Embedding layer, but I am not able to find any more specific information about the actual behavior and training process of this layer. For now, I understand that the ...
2
votes
1answer
45 views

Balanced datasets are almost all predicted negative

Problem I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to ...
0
votes
0answers
30 views

Improve the accuracy of semantic text matching

I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning. I tried several approaches: using ...
4
votes
1answer
257 views

What is the intuition behind the positional cosine encoding in the transformer network?

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence. ...
0
votes
0answers
9 views

Creating a zero element in embedding space

I have some variable length input vectors for my own use case of a 'stylistic transfer'-esque process, and I am wondering if anyone knows of a way to engineer an input that maps to a 0 element in ...
0
votes
0answers
88 views

Word2Vec: Why use input embeddings instead of output embeddings in the skip-gram model?

When using the skip-gram model to learn word embeddings, the algorithm returns two types of embeddings for each word, an input embedding and an output embedding. Usually the input embedding is used ...
0
votes
1answer
185 views

Efficiently normalize word embeddings

I'm using glove word embedding and would like to [-1,1] normalize it using python. The data is in the format of a dict with the word as key and a ...