Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
281
questions
1
vote
0
answers
355
views
fine tune universal-sentence-encoder embeddings
I am new to NLP and Neural Networks. I want to do topic analysis for a dataset of reviews of our product. I tried to use the universal-sentence-encoder along with <...
0
votes
0
answers
119
views
What pretrained word embeddings does the Universal sentence encoder use for Deep Averaging Network?
The paper for Universal sentence encoder Universal sentence encoder paper! is pretty straightforward, and so is the paper for Deep averaging network Deep averaging network paper! but I'm confused ...
0
votes
0
answers
37
views
Continuous Bag of Words NY Time Corpus
I am working to implement the continuous bag of words approach on the New York Times corpus dataset. However, I am getting word embeddings that do not seem very useful based on a few examples of ...
1
vote
0
answers
42
views
Why use a hash bucket to handle Out-of-vocabulary tokens in embedding layers?
For example, in the nnlm-en-dim128 model in thug (https://tfhub.dev/google/nnlm-en-dim128/1). It says
Small fraction of the least frequent tokens and embeddings (~2.5%) are replaced by hash buckets. ...
0
votes
0
answers
304
views
How to train a custom embedding?
I have data with a lot of categorical features. The cardinality of some of these features is quite big (>100), so I want to avoid using one-hot encoding. The idea is to use an embedding.
The ...
4
votes
1
answer
1k
views
How does torch.nn.Embedding or tf.keras.layers.Embedding compare to a dense layer?
Embedding layers are a common choice to map some high-dimensional, discrete input to real-valued (computationally represented using floating point) numbers in a much smaller number of dimensions. Some ...
2
votes
2
answers
314
views
Word Embeddings with nn.Embedding except glove
I'm not an expert in nlp but I know we usually use glove, word2vec or fasttext ect to get embedding vectors so what is nn.Embedding and what does it do?
I mean shouldn't it be specialized for each ...
1
vote
1
answer
1k
views
Difference between token embedding and character embedding in ELMo model
I am learning about a famous NLP model called ELMo. In the explanations, they talk about two types of embeddings. 1) character representations 2) token representations.
Why is there a need to consider ...
1
vote
0
answers
179
views
Single Embedding Layer for Multiple Categorical columns?
I have 100 binary categorical columns to train a neural network model. Each row will be a vector-like [1,1,0,...1] of length 100. I fed this vector to a neural network for a classification problem and ...
1
vote
2
answers
484
views
What makes training time longer with bigger parameter size in a deep learning model?
I try to understand, is it always the case with the more parameter you trained, the more training time you need when training a deep learning model. For example, i have a CNN model for text ...
1
vote
1
answer
1k
views
Understanding number of rnn units in RNN networks
I am trying to learn about recurrent neural networks from here.
There are rnn_units = 1024 in the model and each batch contains ...
1
vote
1
answer
101
views
Ensure trained word embeddings get high similarity with particular words
I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times
...
1
vote
1
answer
610
views
Does not 'removing the stopwords' affect Natural Language Processing Results in a high degree?
Most stopword lists contain contradicting prepositions (before-after, into-out of) and negativity words (not, no). Removing such words from the text almost always changes the meaning drastically.
The ...
1
vote
0
answers
68
views
PCA: removing dominant vector "directions" (isotropy)
I am currently reading an NLP paper on improving the representation of word vectors in space. The authors show that embedded words are not uniformly distributed in space but are contained in a lower-...
1
vote
0
answers
85
views
Definition of the word "embedding"
The mathematical definition of the word "embedding" requires the mapping to be injective, so in that context one speaks of, for example, embedding real numbers in complex numbers (ie, ...
3
votes
1
answer
588
views
How are sentences one-hot encoded internally in an Embedding Keras Layer?
Multiple references are clear on how a single word is one-hot encoded in an Embedding layer, but what about sentences?
In order to illustrate an example, I will use the following SO reference. Let's ...
0
votes
1
answer
484
views
Word2vec/SkipGram: Why softmax?
In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling.
But if in the ...
0
votes
0
answers
29
views
Word-embedding does not provide expected relations between words
I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
2
votes
1
answer
367
views
How does Word2Vec CBOW softmax work with multiple context words?
I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4):
$$
p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
1
vote
0
answers
283
views
Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?
I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer).
However, ...
0
votes
0
answers
80
views
Cross entropy error: Poor modelling giving too much weight to unlikely events
I was reading this paper. link (page 5)
In this paper, there is a statement that goes like this:
To begin, cross entropy error is just one among many possible distance
measures between probability ...
5
votes
1
answer
47
views
Generating embeddings for languages without a written representation?
I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as ...
1
vote
0
answers
19
views
The embeddings of Google document is stemmed or not? [closed]
I will use the embedding google document to do some project. Just want to know whether it was stemmed of the words?
1
vote
0
answers
163
views
Learn sentence embeddings from a sequence of token embeddings
I want to build a sentence classifier that takes the sentence as a sequence of token embeddings. I'm specifically interested in the methodology for learning the sentence embedding from the sequence of ...
1
vote
1
answer
267
views
Extracting word embedding features of a sentence using Transformer-XL
As you know, there are several pre-trained models that we can use to extract word embeddings.
As an example, I can use the following codes to retrieve word2vec features of my text:
...
1
vote
0
answers
65
views
Skipgram model theory confusion
In the output layer of a skipgram model, there are $|\text{Context}|*|\text{Vocab}|$ values. And for each context word, the values are basically the dot product of the input word representation and ...
2
votes
1
answer
2k
views
Distance measure in word2vec
I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation.
I have often read that the ...
1
vote
1
answer
295
views
Intuition for GloVe word embeddings
I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used.
The ...
1
vote
2
answers
413
views
Is there a seq2seq model that can encode sentences that include numerical values?
I am trying to build a seq2seq model that encodes sentences which include numerical values. For example,
Patient's systolic blood pressure was 128.
Conventional ...
0
votes
0
answers
20
views
Choosing a model for input: categorised, weighted sequence, output: binary variable
What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
1
vote
1
answer
453
views
What's the best practice for dealing with OOV characters?
I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
4
votes
1
answer
72
views
Why do we not use continuously defined losses in NLP?
I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors ...
2
votes
1
answer
90
views
Debiasing word embeddings
I'm reading the paper titled "man to computer programmer is woman to homemaker. Debiasing word embeddings". I'm right now trying to figure out the math and logic behind it and was doing OK ...
4
votes
1
answer
439
views
Which Method, Skip-gram or Negative-sampling, does Keras's Embedding Layer Use?
I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
0
votes
1
answer
626
views
Understanding Training Word Embeddings?
I am new to Natural Language Processing. I am trying to understand how word embeddings are created.
When we are training Neural Networks, it is usuallly the weights of the neural network that are ...
1
vote
1
answer
1k
views
When should an embedding layer be used? How big should an embedding be?
I am currently in the process of learning about seq2seq autoencoders for a task involving sentence embedding (samples are sentences, with words represented as integers in a vocab of size $n$). In the ...
3
votes
1
answer
1k
views
Word embeddings - Pre-trained tokenizers vs more involved methods
I'm drowning under all the various methods of converting my text corpora into embeddings.
I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using ...
3
votes
0
answers
1k
views
Different ways to calculate pointwise mutual information for word co-occurrence [closed]
I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count.
I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
0
votes
0
answers
19
views
Feeding Word Embeddings into Recurrent Neural Networks [duplicate]
I'm trying to understand how Recurrent Neural Networks use word embedding vectors as their inputs, and I've created the illustration below to reflect my current-state understanding.
I understand that ...
3
votes
1
answer
675
views
Feeding Word Embeddings into Recurrent Neural Networks
I'm trying to understand how Recurrent Neural Networks use word embedding vectors as their inputs, and I've created the illustration below to reflect my current-state understanding.
I understand that ...
10
votes
1
answer
1k
views
Why does BERT has a limitation of only allowing the maximum length of the input tokens as 512?
I have seen BERT was one of the state-of-the-arts word embedding method in 2018 and then XLNet is proposed in 2019 to take care of the limitations of BERT. I have seen one limitation of BERT is the ...
1
vote
1
answer
228
views
How to replace scalars with vectors in simple models, such as classification of sentences where 1-hot encoding is replaced with word vectors
I have a problem, which seems simple enough, but I don't know how it is solved in the industry. This has to do with the machinery of feeding data to a model, rather than trying to figure out the best ...
2
votes
0
answers
617
views
What is GloVe model's loss function actually minimizing?
I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
0
votes
1
answer
409
views
Use of the term 'dimension' in word-embeddings or various other tensors in AI
I've noticed that AI community refers to various tensors as 512-d, meaning 512 dimensional tensor, where the term 'dimension' seems to mean 512 different float values in the representation for a ...
1
vote
2
answers
108
views
Why are character level models considered less effective than word level models?
I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
6
votes
2
answers
5k
views
When computing parameters, why is dimensions of hidden-output state of an LSTM-cell assumed same as the number of LSTM-cell?
I was trying to figure out how to estimate the number of parameters in an LSTM layer. What is the relationship of number of parameters with the num lstm-cells, input-dimension, and hidden output-state ...
1
vote
0
answers
36
views
Multiple input neural network?
I'm currently working on a social media analysis and trying to predict the tendency to a healthy lifestyle by a social media profile from Vkontakte.
There are several papers on this topic on the web, ...
1
vote
1
answer
1k
views
Can Latent Dirichlet Allocation (LDA) be used to generate word embeddings?
In the original Word2Vec paper (Efficient Estimation of Word Representations in Vector Space, Mikolov et al. 2013), I came across this phrase:
Many different types of models were proposed for ...
2
votes
1
answer
1k
views
Generating Sentence Vectors from Word2Vec
I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec.
I read lot of materials and found that Averaging the ...
4
votes
1
answer
2k
views
Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?
I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine):
Note that (as is usually the case in Machine Learning) we ...