Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0
votes
2answers
7 views

Is there a seq2seq model that can encode sentences that include numerical values?

I am trying to build a seq2seq model that encodes sentences which include numerical values. For example, Patient's systolic blood pressure was 128. Conventional ...
0
votes
0answers
15 views

Negative Sampling in Word2Vec - Embedding Vector / Amount of Samples

I understand that negative sampling in the skip-gram model of word2vec changing the classification from ...
0
votes
0answers
16 views

Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
0
votes
0answers
7 views

How to built vocabulary of numeric words comprised of cells using MatLab?

I have two different variables. First variable is comprised of 104x1 cells and second variable is comprised of 160x240 cells. I have to develop a dictionary that contains both of these variables. How ...
0
votes
0answers
12 views

determining the association of two words based on their word vectors

Suppose I have 2 sets of words: 1- (A, B) 2- (C, D) I have obtained the word vectors of these four words through word embedding. I want to determine if the two words in each set are either strongly ...
0
votes
1answer
19 views

What's the best practice for dealing with OOV characters?

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
0
votes
0answers
9 views

Comparison of stemmed and unstemmed word embeddings (W2V)?

I have a corpus of 10,000 documents. Now I create two wordembedding spaces using a W2V model: I first stem all words in the corpus and then train a W2V model on it. I train the W2V model on the ...
4
votes
1answer
39 views

Why do we not use continuously defined losses in NLP?

I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors ...
0
votes
0answers
4 views

Open source pre-trained models for taxonomy/general word classification

given any two words I'd like to understand if there's some sort of taxonomy/semantic field based relationship. For example given the words "Dog" and "Cat" I'd like to have a model ...
2
votes
1answer
30 views

Debiasing word embeddings

I'm reading the paper titled "man to computer programmer is woman to homemaker. Debiasing word embeddings". I'm right now trying to figure out the math and logic behind it and was doing OK ...
0
votes
0answers
15 views

Best Way to Get Flattened Sentence Embeddings using Individual Word Embeddings - Glove/Embedding Layer Keras

Okay so basically I have a dense matrix of sentence embeddings within which each word in the sentence is embedded to a dimension of (1 x 100). Sentence embeddings with word embeddings of shape (1 x 2) ...
1
vote
1answer
51 views

Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?

I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
0
votes
0answers
8 views

Understanding Training Word Embeddings?

I am new to Natural Language Processing. I am trying to understand how word embeddings are created. When we are training Neural Networks, it is usuallly the weights of the neural network that are ...
1
vote
1answer
50 views

When should an embedding layer be used? How big should an embedding be?

I am currently in the process of learning about seq2seq autoencoders for a task involving sentence embedding (samples are sentences, with words represented as integers in a vocab of size $n$). In the ...
0
votes
1answer
80 views

Word embeddings - Pre-trained tokenizers vs more involved methods

I'm drowning under all the various methods of converting my text corpora into embeddings. I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using ...
1
vote
0answers
94 views

Different ways to calculate pointwise mutual information for word co-occurrence

I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count. I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
0
votes
0answers
12 views

Probabilistic window for negative sampling?

Simple question about skip-grams/negative sampling. When negative sampling, usually a window is created of size 2-10 to define words that are "similar." This is a big improvement over some ...
0
votes
0answers
8 views

Feeding Word Embeddings into Recurrent Neural Networks [duplicate]

I'm trying to understand how Recurrent Neural Networks use word embedding vectors as their inputs, and I've created the illustration below to reflect my current-state understanding. I understand that ...
2
votes
1answer
19 views

Feeding Word Embeddings into Recurrent Neural Networks

I'm trying to understand how Recurrent Neural Networks use word embedding vectors as their inputs, and I've created the illustration below to reflect my current-state understanding. I understand that ...
0
votes
0answers
22 views

product categorization by short strings and one or more powerful factors - a climbers problem

I am using webscraping data to classify products related to the sport of climbing. For all shops i get a product name (which is kind of a short description) plus a category-string as how the product ...
1
vote
0answers
157 views

Why does BERT has a limitation of only allowing the maximum length of the input tokens as 512?

I have seen BERT was one of the state-of-the-arts word embedding method in 2018 and then XLNet is proposed in 2019 to take care of the limitations of BERT. I have seen one limitation of BERT is the ...
0
votes
0answers
7 views

When training embeddings should negative samples be distinct from the context?

Suppose I am training word2vec embeddings with skipgrams. I have defined my context and my target word, and now I am looking for negative samples. It just so happens that I randomly sample a word that ...
0
votes
0answers
12 views

Doc2Vec model training using arbitrary document index or classification category

I'm using gensim's doc2vec to classify news articles into 3 categories(positive, negative, and neutral). I saw a few examples on the web, but I don't quite understand how the document tagging should ...
1
vote
1answer
16 views

How to replace scalars with vectors in simple models, such as classification of sentences where 1-hot encoding is replaced with word vectors

I have a problem, which seems simple enough, but I don't know how it is solved in the industry. This has to do with the machinery of feeding data to a model, rather than trying to figure out the best ...
0
votes
0answers
55 views

What is GloVe model's loss function actually minimizing?

I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
0
votes
1answer
49 views

Use of the term 'dimension' in word-embeddings or various other tensors in AI

I've noticed that AI community refers to various tensors as 512-d, meaning 512 dimensional tensor, where the term 'dimension' seems to mean 512 different float values in the representation for a ...
1
vote
2answers
31 views

Why are character level models considered less effective than word level models?

I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
0
votes
0answers
13 views

Keras LSTM POS tagger w/ transfer learning (GloVe) — failing to learn?

I've been trying to research how to use Keras to train a POS tagger; specifically I want it to use an LSTM architecture and to use word embeddings, namely, GloVe. I've taken inspiration from two blogs....
0
votes
0answers
14 views

size of LSTM layer and regression on fraction

I need to give prediction on a variable with a large range of values (all positives). I scaled the values between 0 and 1. My first layer is an embedding layer, which it's vocabulary size might change....
1
vote
1answer
378 views

When computing parameters, why is dimensions of hidden-output state of an LSTM-cell assumed same as the number of LSTM-cell?

I was trying to figure out how to estimate the number of parameters in an LSTM layer. What is the relationship of number of parameters with the num lstm-cells, input-dimension, and hidden output-state ...
0
votes
0answers
14 views

Number of bidirectional LSTMs in encoder-decoder model must equal the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
1
vote
0answers
24 views

Multiple input neural network?

I'm currently working on a social media analysis and trying to predict the tendency to a healthy lifestyle by a social media profile from Vkontakte. There are several papers on this topic on the web, ...
1
vote
1answer
70 views

Can Latent Dirichlet Allocation (LDA) be used to generate word embeddings?

In the original Word2Vec paper (Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013), I came across this phrase: Many different types of models were proposed for ...
1
vote
1answer
263 views

Generating Sentence Vectors from Word2Vec

I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec. I read lot of materials and found that Averaging the ...
1
vote
1answer
104 views

Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?

I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine): Note that (as is usually the case in Machine Learning) we ...
0
votes
0answers
6 views

Where the embeddings should be implemented in the RNN model?

Hi All (it's my first question here so welcome everyone), I wrote simple RNN model in tensor flow and I cannot figure out where the embeddings should be inserted inside, please find my code and below ...
0
votes
1answer
47 views

Predicting correct match for short French and English descriptions

I have a training and test set of food descriptions pairs (please, see example below) First name in a pair is a name of food in French and second word is this food description in English. Traing set ...
1
vote
1answer
35 views

Downweight or partially mask certain inputs to Neural Network

I have an NLP classification task for sentences set up, in which the goal is to predict a sentence label that depends on the primary verb used in the sentence. This task can be solved by just ...
1
vote
1answer
23 views

Open source resources for transformer based language model and document embeddings

I'm looking for an out of the box transformer model that I can use that can give me document vectors for a list of text. I've looked at some of the BERT like transformers from huggingface but am ...
0
votes
1answer
12 views

Training embedding model on entire corpus then classifying documents from that corpus using trained embeddings

Let's say I have a large corpus of documents. Instead of using a pretrained embedding model, I train my own non-contextual embedding model like w2v/fasttext from scratch on all the documents and save ...
1
vote
1answer
50 views

What are the probabilities in the embedding layer of a Word2Vec?

I am trying to understand how a Word2Vec is being trained. I understand that it can be trained using a CBOW and SkipGram. I am however lost as to what the probabilities are in the embedding layer. ...
0
votes
0answers
10 views

Discriminator loss (SDDE) collapsing in an unexpected way

I'm trying to implement this paper: https://www.aclweb.org/anthology/N19-1255/ During the training process, the task is to encode the sentences into embedding and use those embedding to create the ...
0
votes
0answers
19 views

Embedding from sequence of actions

I have 100 sequences of the word (i.e., action for completing a task). Each of the sequences contains around 350 actions(115 unique actions but all the actions are not used in each sequence. Some of ...
2
votes
2answers
718 views

Difference between non-contextual and contextual word embeddings

It is often stated that word2vec and GloVe are non-contextual embeddings while LSTM and Transformer-based (e.g. BERT) embeddings are contextual. The way I understand it, however, all word embeddings ...
0
votes
0answers
36 views

What to make of high R-squared and non-significant p-value of a linear model?

I am using doc2vec to produce $\mathbb{R}^{50}$ vector representations of short bits of text. I am then using those vectors in a linear model to predict a continuous outcome variable. The R^2 is .25 ...
1
vote
1answer
50 views

What is the target variable in the feed forward neural network within the Transformer model architecture?

In the paper 'Attention is all you need' the model architecture of The Transformer is described. Both in the encoder as well as in the decoder, there is a feed forward network. If I understant it ...
0
votes
1answer
332 views

Train a RNN with unknown vocabulary size

I'm new to deep learning and i'm trying to code a Visual Question answering network. I studied and (i think) understood how RNN and LSTM work. From what i'he understood, i need to train my network ...
1
vote
0answers
32 views

How does using the same vector for the center word and for the context word impact the performance of word vectors in word2vec?

By default, word2vec uses 2 vectors for each word: one for the center word and one for the context word: $\color{steelblue}{\large \text{Word2vec: objective function}}$ $\color{darkred}{\scriptstyle{\...
1
vote
0answers
101 views

FastText or GloVe for code-mixed sentiment analysis?

I am currently working on a project for code-mixed sentiment analysis (English+Spanish). I've been using the GloVe Twitter word embeddings so far but I realized that even though this representation is ...
1
vote
1answer
60 views

How can this L(2,1) problem be reduced to the orthogonal procrustes problem?

NOTE: Don't take this too serious -- the question is actually due to my misreading $\|y_i - Wx_i\|^2$ as $\|y_i - Wx_i\|_2$, see the answer. Smith et al. in Offline bilingual word vectors, orthogonal ...

1
2 3 4 5