Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
283 questions
0
votes
0
answers
12
views
Possible to use TSDAE for word discovery?
I am trying to identify novel words similar to one that I have encoded. From what I have been reading online, the majority of systems which look for word similarity have a predefined list of words and ...
0
votes
0
answers
23
views
Words’ similarity formula based on the context words
I am working on my word embedding calculation algorithm and stuck with a similarity formula.
I assume that this could be easily derived formally with statistics and probabilities, but I fail to do so. ...
- 203
0
votes
0
answers
31
views
Word embeddings with a constrained loss variational autoencoder
I got an idea while refreshing some knowledge about CBOW and Skip-Gram. I thought to post here to receive some criticism and perhaps learn something new.
The idea was to use a basic Variational ...
- 768
1
vote
1
answer
40
views
What's the minimum number of points worth clustering? [closed]
I'm using HDBSCAN to cluster vector embeddings of sentences. The embeddings are being drawn from an online SQL table that's constantly being updated. At peak times, this means 1,000 embeddings an hour,...
- 667
1
vote
1
answer
156
views
How does two tower model map to shared embedding space for two different type of entity?
A canonical example is say you have user and merchandise
user (feature: age, location....)
merchandise (feature: type, size, .....)
And you want to create embedding to map user and merchandise to same ...
- 11
1
vote
0
answers
52
views
Measuring Similarity in Embedding Spaces? [closed]
For context, I've been using feature hashing for a rapid text classifier with a very small number of features (2000, it is very small on purpose). I noticed that some of the results were a bit wonky ...
0
votes
0
answers
38
views
Normalizing the embedding space of an encoder language model with respect to categorical data
Suppose we have a tree/hierarchy of categories (e.g. categories of products in an e-commerce website), each node being assigned a title. Assume that the title of each node is semantically accurate, ...
- 123
0
votes
0
answers
35
views
Why learn an embedding before self attention when training transformers?
I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
6
votes
1
answer
173
views
What advantage do sinusoidal positional encodings have over binary positional encodings in transformer LLMs?
I've recently come across an article that discusses the reasons why large language models use sinusoidal functions to generate positional encodings — as per the famous paper Attention Is All You Need (...
- 156
0
votes
0
answers
13
views
Appropriateness of the Universal Sentence Encoder model
I have a classification problem where the goal is to predict, based on a small paragraph, if an individual is British or not.
The model used for the classification is Universal Sentence Encoder (to ...
- 1
0
votes
0
answers
153
views
Locality sensitive hashing (LSH) with word embeddings and cosine similarity
I would like to ask about the methodology of LSH algorithm with Word Embeddings and Cosine Similarity to identify similar documents.
First, I tokenize my sentences to create a list of tokens. Then, I ...
- 123
0
votes
0
answers
248
views
Is it necessary to use attention mask for mean pooling for BERT?
I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
- 138
1
vote
0
answers
20
views
Team vectors similar to word vectors for a corpus of text
Is there any way I could iteratively create a set of vectors, similar to vectors when embedding words through word2vec, that could represent vectors between teams, and also capture information about a ...
- 11
0
votes
1
answer
139
views
What embeddings are used in decoder-only models like GPT?
Decoder-only models do not use an encoder. Hence, they do not get the embedding from it.
I went through this nice description of a decoder-only transformer -based model
I do understand the training ...
- 55
2
votes
0
answers
50
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 66
0
votes
1
answer
835
views
Are embedding in GPT models trainable model parameters? [closed]
I have tried to search from a few sources, but I did not see any one of them specifically talking about this issue. For example This blog post seems to imply that the embedding used in transformer is ...
- 413
0
votes
1
answer
44
views
Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?
In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
- 31
1
vote
1
answer
120
views
How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?
I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
- 31
0
votes
1
answer
522
views
Using Word Embeddings in Clustering and Topic Modelling
I am new to the field of NLP and would appreciate any guidance please. I am trying to understand how word embeddings can be used in clustering and topic modelling. If I create word embeddings for ...
- 31
0
votes
0
answers
53
views
How is OpenAI embedding obtained
There is OpenAI embedding API
https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input ...
- 519
0
votes
1
answer
442
views
Word embedding and Euclidean distance
Does a transformation exist that allows to use of the Euclidean distance with the word embeddings? The Cosine distance could be a problem in my case.
For example, what if I translate the vector to a ...
- 171
1
vote
1
answer
226
views
How does training word embeddings bring similar words closer together?
How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
- 485
1
vote
2
answers
137
views
Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?
In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
- 485
0
votes
1
answer
69
views
Using embeddings to anonymize information
This might be a stupid question, so bear with me.
I was wondering if embeddings can be used to anonymize input text.
I couldn't find any information online that says that embeddings can be 1:1 decoded ...
- 103
1
vote
0
answers
57
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 11
2
votes
1
answer
49
views
Composed cosine similarity
I have the following problem.
I have 3 vectors $u,v,w$ of n dimensions.
I'm able to find cosine similarities between $u$ and $v$, and between $v$ and $w$: $cosine(u,v)$ and $cosine(v,w)$.
Can i use ...
- 21
6
votes
1
answer
1k
views
How to compare the semantic similarity of text generated by large language models (GPT-3, BLOOM etc) to reference text?
Large language models, such as GPT-3, BLOOM etc, can generate open-ended text. Say I want to prompt these models to answer a question. How can I compare the semantic similarity of the answer it ...
- 61
1
vote
0
answers
67
views
High loss while training word embedding
I'm trying to train (Skip-Gram) word embeddings on a dataset derived from a small piece of data (500 lines ~2500 words ~840 unique words).
I'm think I'm aware that this is too small of a dataset to ...
- 11
0
votes
1
answer
34
views
Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)
I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
- 1,046
1
vote
0
answers
51
views
Is there an analogous calculation for weighted max and min, similar to weighted average, for text embeddings?
Say I have created some numerical embedding for each word in a text corpora -- that is, for each word I have some n-dimensional vector representing the word. Now say I would like to perform some ...
- 308
1
vote
1
answer
1k
views
Do BERT word embeddings change after training, depending on context?
Before answering "yes, of course", let me clarify what I mean:
After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
1
vote
0
answers
231
views
Equivalence between one-dimensional embedding and one-hot encoding?
This may not sound like a practical question, but I am trying to create essentially a toy problem that can help me understand and apply some explainability algorithms (eg Integrated Gradients). To do ...
- 1,965
4
votes
1
answer
4k
views
Training Transformers: self attention weights vs embedding layer
I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
- 143
2
votes
1
answer
372
views
Approaches for semi-supervised fine-tuning after self-supervised pre-training
My understanding is that self-supervised learning approaches approximately work like the following (I have Wav2Vec 2 in my mind here, used in speech recognition, but NLP transformer models are similar)...
- 95
0
votes
1
answer
112
views
How to encode an url into into a 60-bit 01 string? (word2vec) [closed]
At the same time, we counted the frequency of occurrences of
characters in all URLs in the dataset and selected the first 59
characters with the highest frequency as valid characters. It contains
26 ...
2
votes
0
answers
859
views
Alternatives to the Silhouette score in cluster analysis?
The silhouette score in my case gives quite misleading results, any alternatives?
My data is a result of embedding words, which belong to one of the 20+ classes. I want to measure the "...
- 185
0
votes
1
answer
336
views
Why should the weight matrix encode word embeddings in CBOW/skip-gram?
Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings.
I'm looking for an intuitive ...
4
votes
2
answers
2k
views
What exactly is meant by isotropic and anisotropic with word vectors
From this paper https://aclanthology.org/D19-1006.pdf "How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings"
When they say word ...
- 182
1
vote
0
answers
229
views
Skip gram model and negative log loss likelihood
I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
- 23
5
votes
2
answers
724
views
Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding
In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on).
This way, each cell contains semantic and ...
- 361
1
vote
0
answers
39
views
Why do we call word embeddings "vectors" ? Shouldn't we call them word coordinates?
Word embeddings are often referred to as vectors. For instance, "Word embedding models are dense vector representations of words learned from a corpus of natural language" (Rosado, 2020). ...
- 592
1
vote
1
answer
28
views
Algorithm to find closest document containing a set of strings, or variations of them
I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers.
I have another dataset (B) which contains highly ...
- 465
0
votes
1
answer
72
views
Ways to detect noise in multi-class classification training data using text embeddings (BERT)
So I have a dataset with a column of text and and labels (5 different labels) associated with it. The labels describe the potential answer to the type of question being asked in the text column. For ...
1
vote
0
answers
803
views
Cosine Similarity of the word embeddings after UMAP dimensionality reduction
I want to calculate similarities of the word embeddings. As a basis I took SpaCy german corpus: nlp = spacy.load("de_core_news_lg").
I have approximately ...
- 1,452
1
vote
0
answers
65
views
Why does latent dirichlet allocation (LDA) fail when dealing with large and heavy-tailed vocabularies?
I'm reading the 2019 paper Topic Modeling in Embedding Spaces which claims that the embedded topic model improves on these limitations of LDA. But why does LDA have these limitations—why does it fail ...
1
vote
1
answer
2k
views
Interconnections between embeddings layer and LSTM layer
I'm trying to build a text classifier with keras using word embeddings (glove) and a RNN (in this case a LSTM) using keras. I searched in several sites and decided to start with this configuration:
<...
- 11
2
votes
1
answer
58
views
Text similarity for badly written text
Consider the following scenario:
Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
- 337
2
votes
1
answer
649
views
Should I use cosine or dot similarity inside word2vec's neural network?
I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
- 185
4
votes
1
answer
994
views
How is an embedding space optimized with respect to the loss function?
I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in ...
- 679
1
vote
1
answer
340
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
- 41