Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
282
questions
0
votes
0
answers
22
views
How to add multiple embeddings (layers) to LSTM layer [closed]
The similar question was asked before here https://stackoverflow.com/questions/52627739/how-to-merge-numerical-and-embedding-sequential-models-to-treat-categories-in-rn/52629902#...
0
votes
1
answer
50
views
Are embedding in GPT models trainable model parameters? [closed]
I have tried to search from a few sources, but I did not see any one of them specifically talking about this issue. For example This blog post seems to imply that the embedding used in transformer is ...
- 363
0
votes
0
answers
10
views
How can I compute prediction explainability (e.g. SHAP) for a classifier trained on dense embeddings (SBERT)?
I have a multiclass classification problem (intent classification). I trained an XGBoost model on a dataset that was feature extracted using SBERT embeddings. I'm trying to compute an explainability ...
- 159
0
votes
0
answers
3
views
Quantify Independence/Jointness of Tasks in Multitask Model
Suppose I have two identical tasks that are over two related but not identical domains- for example, next word prediction in Hamlet in both English and in Spanish. I want to train a model on a mix of ...
0
votes
0
answers
9
views
How word2vec de-embeds the special names in language models which output text
I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary ...
- 169
0
votes
0
answers
20
views
What are more sophisticated measures for word similarity than cosine similarity?
I am trying to compute measures of word/sentence similarity from word embeddings that I would like to use for classification. However, cosine similarity often runs counter to how I as a human would ...
- 101
0
votes
0
answers
10
views
How does Doc2Vec handle documents where the length is less than the window size?
I am using doc2vec to vectorise documents that average in length 10 words (PV-DBOW implementation of the algorithm). I am wondering how doc2vec handles cases where the number of words in a document is ...
- 31
0
votes
0
answers
10
views
How do Doc2Vec PV-DBOW iterations/epoch's work?
Please correct me if I'm wrong but as I understand it, one iteration (?) of doc2vec takes each document and predicts a series of randomly sampled context words individually, feeding the errors from ...
- 31
0
votes
1
answer
18
views
Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?
In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
- 31
1
vote
1
answer
54
views
How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?
I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
- 31
0
votes
1
answer
54
views
Using Word Embeddings in Clustering and Topic Modelling
I am new to the field of NLP and would appreciate any guidance please. I am trying to understand how word embeddings can be used in clustering and topic modelling. If I create word embeddings for ...
- 31
0
votes
0
answers
35
views
How is OpenAI embedding obtained
There is OpenAI embedding API
https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input ...
- 459
0
votes
1
answer
71
views
Word embedding and Euclidean distance
Does a transformation exist that allows to use of the Euclidean distance with the word embeddings? The Cosine distance could be a problem in my case.
For example, what if I translate the vector to a ...
- 171
0
votes
1
answer
52
views
How does training word embeddings bring similar words closer together?
How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
- 433
0
votes
1
answer
39
views
Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?
In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
- 433
0
votes
1
answer
40
views
Using embeddings to anonymize information
This might be a stupid question, so bear with me.
I was wondering if embeddings can be used to anonymize input text.
I couldn't find any information online that says that embeddings can be 1:1 decoded ...
- 103
0
votes
0
answers
8
views
How can I produce word embeddings from code-mixed data using Facebook MUSE?
I am doing sarcasm detection using code mixed data. Sentences are in English and Hindi written in English alphabets. Is there a way to apply Facebook MUSE to generate word embeddings out of this data?
- 119
1
vote
0
answers
36
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 11
2
votes
1
answer
23
views
Composed cosine similarity
I have the following problem.
I have 3 vectors $u,v,w$ of n dimensions.
I'm able to find cosine similarities between $u$ and $v$, and between $v$ and $w$: $cosine(u,v)$ and $cosine(v,w)$.
Can i use ...
- 21
3
votes
1
answer
590
views
How to compare the semantic similarity of text generated by large language models (GPT-3, BLOOM etc) to reference text?
Large language models, such as GPT-3, BLOOM etc, can generate open-ended text. Say I want to prompt these models to answer a question. How can I compare the semantic similarity of the answer it ...
- 31
1
vote
0
answers
32
views
High loss while training word embedding
I'm trying to train (Skip-Gram) word embeddings on a dataset derived from a small piece of data (500 lines ~2500 words ~840 unique words).
I'm think I'm aware that this is too small of a dataset to ...
- 11
0
votes
1
answer
26
views
Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)
I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
- 1,014
0
votes
0
answers
13
views
Calculating noise distribution in skip gram negative sampling
I was referring to this article explaining skip-gram with negative sampling. It says we need to sample negative samples from noise distribution calculated as follows:
$$P_n(w) = \left(\frac{U(w)}{Z}\...
- 215
0
votes
0
answers
53
views
BERT embeddings are different in batches, compared to single input
I'm using BERT embeddings in an SVM. For computational efficiency, I'm batching my data, feeding them into BERT to get the embeddings from the last 4 hidden layers, and then using these embeddings in ...
- 31
0
votes
0
answers
30
views
Running SVM on positional embeddings using keras for text classification
How can I run SVM on a large text classification dataset for detecting fake news of 400 thousand entries that uses positional encoding for embeddings from keras and has a maximum sentence length of 15 ...
- 1
1
vote
0
answers
36
views
Is there an analogous calculation for weighted max and min, similar to weighted average, for text embeddings?
Say I have created some numerical embedding for each word in a text corpora -- that is, for each word I have some n-dimensional vector representing the word. Now say I would like to perform some ...
- 298
0
votes
0
answers
59
views
How to best compute pairwise similarity vectors with additional redundancy consideration
My goal is to compute pairwise similarity between 700 regions (i.e., columns of Dataframe #1 described below); hence a 700x700 output matrix.
I have two dataframes I would like to consider in the ...
0
votes
0
answers
20
views
Which pre-trained model to use to generate embeddings from shop names?
Good afternoon!
I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
- 93
1
vote
1
answer
523
views
Do BERT word embeddings change after training, depending on context?
Before answering "yes, of course", let me clarify what I mean:
After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
1
vote
0
answers
137
views
Equivalence between one-dimensional embedding and one-hot encoding?
This may not sound like a practical question, but I am trying to create essentially a toy problem that can help me understand and apply some explainability algorithms (eg Integrated Gradients). To do ...
- 1,875
1
vote
1
answer
1k
views
Training Transformers: self attention weights vs embedding layer
I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
- 113
2
votes
1
answer
229
views
Approaches for semi-supervised fine-tuning after self-supervised pre-training
My understanding is that self-supervised learning approaches approximately work like the following (I have Wav2Vec 2 in my mind here, used in speech recognition, but NLP transformer models are similar)...
- 95
0
votes
1
answer
76
views
How to encode an url into into a 60-bit 01 string? (word2vec) [closed]
At the same time, we counted the frequency of occurrences of
characters in all URLs in the dataset and selected the first 59
characters with the highest frequency as valid characters. It contains
26 ...
0
votes
0
answers
24
views
Multivariate Gaussian distribution for Word Embeddings
I am creating mock-data for tests for word-embedding operations. If I treat embeddings as samples from a Multivariate Gaussian distribution: are there any false assumptions I am making?
If so what ...
- 115
2
votes
0
answers
390
views
Alternatives to the Silhouette score in cluster analysis?
The silhouette score in my case gives quite misleading results, any alternatives?
My data is a result of embedding words, which belong to one of the 20+ classes. I want to measure the "...
- 75
0
votes
1
answer
227
views
Why should the weight matrix encode word embeddings in CBOW/skip-gram?
Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings.
I'm looking for an intuitive ...
4
votes
2
answers
1k
views
What exactly is meant by isotropic and anisotropic with word vectors
From this paper https://aclanthology.org/D19-1006.pdf "How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings"
When they say word ...
- 152
1
vote
0
answers
121
views
Skip gram model and negative log loss likelihood
I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
- 23
4
votes
1
answer
312
views
Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding
In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on).
This way, each cell contains semantic and ...
- 351
1
vote
0
answers
33
views
Why do we call word embeddings "vectors" ? Shouldn't we call them word coordinates?
Word embeddings are often referred to as vectors. For instance, "Word embedding models are dense vector representations of words learned from a corpus of natural language" (Rosado, 2020). ...
- 552
1
vote
1
answer
21
views
Algorithm to find closest document containing a set of strings, or variations of them
I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers.
I have another dataset (B) which contains highly ...
- 385
0
votes
1
answer
51
views
Ways to detect noise in multi-class classification training data using text embeddings (BERT)
So I have a dataset with a column of text and and labels (5 different labels) associated with it. The labels describe the potential answer to the type of question being asked in the text column. For ...
1
vote
0
answers
544
views
Cosine Similarity of the word embeddings after UMAP dimensionality reduction
I want to calculate similarities of the word embeddings. As a basis I took SpaCy german corpus: nlp = spacy.load("de_core_news_lg").
I have approximately ...
- 1,412
1
vote
0
answers
52
views
Why does latent dirichlet allocation (LDA) fail when dealing with large and heavy-tailed vocabularies?
I'm reading the 2019 paper Topic Modeling in Embedding Spaces which claims that the embedded topic model improves on these limitations of LDA. But why does LDA have these limitations—why does it fail ...
1
vote
1
answer
2k
views
Interconnections between embeddings layer and LSTM layer
I'm trying to build a text classifier with keras using word embeddings (glove) and a RNN (in this case a LSTM) using keras. I searched in several sites and decided to start with this configuration:
<...
- 11
2
votes
1
answer
51
views
Text similarity for badly written text
Consider the following scenario:
Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
- 287
2
votes
1
answer
399
views
Should I use cosine or dot similarity inside word2vec's neural network?
I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
- 75
4
votes
1
answer
611
views
How is an embedding space optimized with respect to the loss function?
I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in ...
- 679
0
votes
1
answer
214
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
- 21
0
votes
0
answers
1k
views
Using BERT Embeddings + Standard ML for text classification
I am trying to automatically detect whether a text is written by a Machine or a Human.
My first approach was using a TF-IDF to build features for a logistic regression classifier, where I got an ...
- 123