Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
1 vote
0 answers
355 views

fine tune universal-sentence-encoder embeddings

I am new to NLP and Neural Networks. I want to do topic analysis for a dataset of reviews of our product. I tried to use the universal-sentence-encoder along with <...
Ahmed Elashry's user avatar
0 votes
0 answers
119 views

What pretrained word embeddings does the Universal sentence encoder use for Deep Averaging Network?

The paper for Universal sentence encoder Universal sentence encoder paper! is pretty straightforward, and so is the paper for Deep averaging network Deep averaging network paper! but I'm confused ...
hamza -'s user avatar
  • 23
0 votes
0 answers
37 views

Continuous Bag of Words NY Time Corpus

I am working to implement the continuous bag of words approach on the New York Times corpus dataset. However, I am getting word embeddings that do not seem very useful based on a few examples of ...
dzheng1887's user avatar
1 vote
0 answers
42 views

Why use a hash bucket to handle Out-of-vocabulary tokens in embedding layers?

For example, in the nnlm-en-dim128 model in thug (https://tfhub.dev/google/nnlm-en-dim128/1). It says Small fraction of the least frequent tokens and embeddings (~2.5%) are replaced by hash buckets. ...
gdlamp's user avatar
  • 131
0 votes
0 answers
304 views

How to train a custom embedding?

I have data with a lot of categorical features. The cardinality of some of these features is quite big (>100), so I want to avoid using one-hot encoding. The idea is to use an embedding. The ...
DocDriven's user avatar
  • 343
4 votes
1 answer
1k views

How does torch.nn.Embedding or tf.keras.layers.Embedding compare to a dense layer?

Embedding layers are a common choice to map some high-dimensional, discrete input to real-valued (computationally represented using floating point) numbers in a much smaller number of dimensions. Some ...
Sycorax's user avatar
  • 90.8k
2 votes
2 answers
314 views

Word Embeddings with nn.Embedding except glove

I'm not an expert in nlp but I know we usually use glove, word2vec or fasttext ect to get embedding vectors so what is nn.Embedding and what does it do? I mean shouldn't it be specialized for each ...
Farhang Amaji's user avatar
1 vote
1 answer
1k views

Difference between token embedding and character embedding in ELMo model

I am learning about a famous NLP model called ELMo. In the explanations, they talk about two types of embeddings. 1) character representations 2) token representations. Why is there a need to consider ...
Kadaj13's user avatar
  • 395
1 vote
0 answers
179 views

Single Embedding Layer for Multiple Categorical columns?

I have 100 binary categorical columns to train a neural network model. Each row will be a vector-like [1,1,0,...1] of length 100. I fed this vector to a neural network for a classification problem and ...
Sanjay's user avatar
  • 163
1 vote
2 answers
484 views

What makes training time longer with bigger parameter size in a deep learning model?

I try to understand, is it always the case with the more parameter you trained, the more training time you need when training a deep learning model. For example, i have a CNN model for text ...
thenoirlatte's user avatar
1 vote
1 answer
1k views

Understanding number of rnn units in RNN networks

I am trying to learn about recurrent neural networks from here. There are rnn_units = 1024 in the model and each batch contains ...
Kadaj13's user avatar
  • 395
1 vote
1 answer
101 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...
Clock Slave's user avatar
  • 1,067
1 vote
1 answer
610 views

Does not 'removing the stopwords' affect Natural Language Processing Results in a high degree?

Most stopword lists contain contradicting prepositions (before-after, into-out of) and negativity words (not, no). Removing such words from the text almost always changes the meaning drastically. The ...
Xfce4's user avatar
  • 129
1 vote
0 answers
68 views

PCA: removing dominant vector "directions" (isotropy)

I am currently reading an NLP paper on improving the representation of word vectors in space. The authors show that embedded words are not uniformly distributed in space but are contained in a lower-...
displayname's user avatar
1 vote
0 answers
85 views

Definition of the word "embedding"

The mathematical definition of the word "embedding" requires the mapping to be injective, so in that context one speaks of, for example, embedding real numbers in complex numbers (ie, ...
Lei Huang's user avatar
  • 956
3 votes
1 answer
588 views

How are sentences one-hot encoded internally in an Embedding Keras Layer?

Multiple references are clear on how a single word is one-hot encoded in an Embedding layer, but what about sentences? In order to illustrate an example, I will use the following SO reference. Let's ...
Fernando Wittmann's user avatar
0 votes
1 answer
484 views

Word2vec/SkipGram: Why softmax?

In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling. But if in the ...
guest's user avatar
  • 103
0 votes
0 answers
29 views

Word-embedding does not provide expected relations between words

I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
user320342's user avatar
2 votes
1 answer
367 views

How does Word2Vec CBOW softmax work with multiple context words?

I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4): $$ p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
jonas's user avatar
  • 123
1 vote
0 answers
283 views

Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?

I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer). However, ...
Charlie Parker's user avatar
0 votes
0 answers
80 views

Cross entropy error: Poor modelling giving too much weight to unlikely events

I was reading this paper. link (page 5) In this paper, there is a statement that goes like this: To begin, cross entropy error is just one among many possible distance measures between probability ...
user27286's user avatar
  • 289
5 votes
1 answer
47 views

Generating embeddings for languages without a written representation?

I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as ...
Skander H.'s user avatar
  • 11.9k
1 vote
0 answers
19 views

The embeddings of Google document is stemmed or not? [closed]

I will use the embedding google document to do some project. Just want to know whether it was stemmed of the words?
Rockko joke's user avatar
1 vote
0 answers
163 views

Learn sentence embeddings from a sequence of token embeddings

I want to build a sentence classifier that takes the sentence as a sequence of token embeddings. I'm specifically interested in the methodology for learning the sentence embedding from the sequence of ...
D Hudson's user avatar
  • 111
1 vote
1 answer
267 views

Extracting word embedding features of a sentence using Transformer-XL

As you know, there are several pre-trained models that we can use to extract word embeddings. As an example, I can use the following codes to retrieve word2vec features of my text: ...
Kadaj13's user avatar
  • 395
1 vote
0 answers
65 views

Skipgram model theory confusion

In the output layer of a skipgram model, there are $|\text{Context}|*|\text{Vocab}|$ values. And for each context word, the values are basically the dot product of the input word representation and ...
Aditya Agarwal's user avatar
2 votes
1 answer
2k views

Distance measure in word2vec

I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation. I have often read that the ...
Josef's user avatar
  • 131
1 vote
1 answer
295 views

Intuition for GloVe word embeddings

I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used. The ...
calveeen's user avatar
  • 1,126
1 vote
2 answers
413 views

Is there a seq2seq model that can encode sentences that include numerical values?

I am trying to build a seq2seq model that encodes sentences which include numerical values. For example, Patient's systolic blood pressure was 128. Conventional ...
Sardar's user avatar
  • 11
0 votes
0 answers
20 views

Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
Ian's user avatar
  • 101
1 vote
1 answer
453 views

What's the best practice for dealing with OOV characters?

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
gmedina-v's user avatar
4 votes
1 answer
72 views

Why do we not use continuously defined losses in NLP?

I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors ...
infobox's user avatar
  • 43
2 votes
1 answer
90 views

Debiasing word embeddings

I'm reading the paper titled "man to computer programmer is woman to homemaker. Debiasing word embeddings". I'm right now trying to figure out the math and logic behind it and was doing OK ...
mitra mirshafiee's user avatar
4 votes
1 answer
439 views

Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?

I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
etang's user avatar
  • 957
0 votes
1 answer
626 views

Understanding Training Word Embeddings?

I am new to Natural Language Processing. I am trying to understand how word embeddings are created. When we are training Neural Networks, it is usuallly the weights of the neural network that are ...
Anirban Saha's user avatar
1 vote
1 answer
1k views

When should an embedding layer be used? How big should an embedding be?

I am currently in the process of learning about seq2seq autoencoders for a task involving sentence embedding (samples are sentences, with words represented as integers in a vocab of size $n$). In the ...
J. Auon's user avatar
  • 139
3 votes
1 answer
1k views

Word embeddings - Pre-trained tokenizers vs more involved methods

I'm drowning under all the various methods of converting my text corpora into embeddings. I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using ...
ML_Engine's user avatar
  • 205
3 votes
0 answers
1k views

Different ways to calculate pointwise mutual information for word co-occurrence [closed]

I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count. I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
exfalso's user avatar
  • 133
0 votes
0 answers
19 views

Feeding Word Embeddings into Recurrent Neural Networks [duplicate]

I'm trying to understand how Recurrent Neural Networks use word embedding vectors as their inputs, and I've created the illustration below to reflect my current-state understanding. I understand that ...
bshelt141's user avatar
  • 227
3 votes
1 answer
675 views

Feeding Word Embeddings into Recurrent Neural Networks

I'm trying to understand how Recurrent Neural Networks use word embedding vectors as their inputs, and I've created the illustration below to reflect my current-state understanding. I understand that ...
bshelt141's user avatar
  • 227
10 votes
1 answer
1k views

Why does BERT has a limitation of only allowing the maximum length of the input tokens as 512?

I have seen BERT was one of the state-of-the-arts word embedding method in 2018 and then XLNet is proposed in 2019 to take care of the limitations of BERT. I have seen one limitation of BERT is the ...
RockTheStar's user avatar
  • 12.9k
1 vote
1 answer
228 views

How to replace scalars with vectors in simple models, such as classification of sentences where 1-hot encoding is replaced with word vectors

I have a problem, which seems simple enough, but I don't know how it is solved in the industry. This has to do with the machinery of feeding data to a model, rather than trying to figure out the best ...
Shahbaz's user avatar
  • 113
2 votes
0 answers
617 views

What is GloVe model's loss function actually minimizing?

I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
Fraïssé's user avatar
  • 1,540
0 votes
1 answer
409 views

Use of the term 'dimension' in word-embeddings or various other tensors in AI

I've noticed that AI community refers to various tensors as 512-d, meaning 512 dimensional tensor, where the term 'dimension' seems to mean 512 different float values in the representation for a ...
Joe Black's user avatar
  • 319
1 vote
2 answers
108 views

Why are character level models considered less effective than word level models?

I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
yash karbhari's user avatar
6 votes
2 answers
5k views

When computing parameters, why is dimensions of hidden-output state of an LSTM-cell assumed same as the number of LSTM-cell?

I was trying to figure out how to estimate the number of parameters in an LSTM layer. What is the relationship of number of parameters with the num lstm-cells, input-dimension, and hidden output-state ...
Joe Black's user avatar
  • 319
1 vote
0 answers
36 views

Multiple input neural network?

I'm currently working on a social media analysis and trying to predict the tendency to a healthy lifestyle by a social media profile from Vkontakte. There are several papers on this topic on the web, ...
nutcracker's user avatar
1 vote
1 answer
1k views

Can Latent Dirichlet Allocation (LDA) be used to generate word embeddings?

In the original Word2Vec paper (Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013), I came across this phrase: Many different types of models were proposed for ...
NeuronQ's user avatar
  • 113
2 votes
1 answer
1k views

Generating Sentence Vectors from Word2Vec

I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec. I read lot of materials and found that Averaging the ...
chaitanya's user avatar
  • 131
4 votes
1 answer
2k views

Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?

I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine): Note that (as is usually the case in Machine Learning) we ...
stackoverflowuser2010's user avatar