Skip to main content

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Normalizing the embedding space of an encoder language model with respect to categorical data

Suppose we have a tree/hierarchy of categories (e.g. categories of products in an e-commerce website), each node being assigned a title. Assume that the title of each node is semantically accurate. ...
mtcicero's user avatar
  • 123
0 votes
0 answers
9 views

Why learn an embedding before self attention when training transformers?

I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
Nicolas Johnson's user avatar
4 votes
0 answers
80 views

What advantage do sinusoidal positional encodings have over binary positional encodings in transformer LLMs?

I've recently come across an article that discusses the reasons why large language models use sinusoidal functions to generate positional encodings — as per the famous paper Attention Is All You Need (...
Philip Voinea's user avatar
0 votes
0 answers
10 views

Appropriateness of the Universal Sentence Encoder model

I have a classification problem where the goal is to predict, based on a small paragraph, if an individual is British or not. The model used for the classification is Universal Sentence Encoder (to ...
Sara Mun's user avatar
0 votes
0 answers
95 views

Locality sensitive hashing (LSH) with word embeddings and cosine similarity

I would like to ask about the methodology of LSH algorithm with Word Embeddings and Cosine Similarity to identify similar documents. First, I tokenize my sentences to create a list of tokens. Then, I ...
BDEngineer's user avatar
0 votes
0 answers
127 views

Is it necessary to use attention mask for mean pooling for BERT?

I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
mutli-arm-bandit's user avatar
1 vote
0 answers
19 views

Team vectors similar to word vectors for a corpus of text

Is there any way I could iteratively create a set of vectors, similar to vectors when embedding words through word2vec, that could represent vectors between teams, and also capture information about a ...
troy3.14's user avatar
0 votes
1 answer
86 views

What embeddings are used in decoder-only models like GPT?

Decoder-only models do not use an encoder. Hence, they do not get the embedding from it. I went through this nice description of a decoder-only transformer -based model I do understand the training ...
tintin98's user avatar
2 votes
0 answers
48 views

How was the word2vec model trained?

Let's take the CBOW (continuous bag of words) model as the example. Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
J. Doe's user avatar
  • 66
0 votes
1 answer
633 views

Are embedding in GPT models trainable model parameters? [closed]

I have tried to search from a few sources, but I did not see any one of them specifically talking about this issue. For example This blog post seems to imply that the embedding used in transformer is ...
Sam's user avatar
  • 403
0 votes
0 answers
47 views

How can I compute prediction explainability (e.g. SHAP) for a classifier trained on dense embeddings (SBERT)?

I have a multiclass classification problem (intent classification). I trained an XGBoost model on a dataset that was feature extracted using SBERT embeddings. I'm trying to compute an explainability ...
Metrician's user avatar
  • 169
0 votes
0 answers
3 views

Quantify Independence/Jointness of Tasks in Multitask Model

Suppose I have two identical tasks that are over two related but not identical domains- for example, next word prediction in Hamlet in both English and in Spanish. I want to train a model on a mix of ...
Garrodactyl's user avatar
0 votes
0 answers
35 views

What are more sophisticated measures for word similarity than cosine similarity?

I am trying to compute measures of word/sentence similarity from word embeddings that I would like to use for classification. However, cosine similarity often runs counter to how I as a human would ...
dufei's user avatar
  • 101
0 votes
0 answers
28 views

How does Doc2Vec handle documents where the length is less than the window size?

I am using doc2vec to vectorise documents that average in length 10 words (PV-DBOW implementation of the algorithm). I am wondering how doc2vec handles cases where the number of words in a document is ...
osckt's user avatar
  • 31
0 votes
1 answer
32 views

Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?

In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
osckt's user avatar
  • 31
1 vote
1 answer
98 views

How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?

I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
osckt's user avatar
  • 31
0 votes
1 answer
461 views

Using Word Embeddings in Clustering and Topic Modelling

I am new to the field of NLP and would appreciate any guidance please. I am trying to understand how word embeddings can be used in clustering and topic modelling. If I create word embeddings for ...
osckt's user avatar
  • 31
0 votes
0 answers
42 views

How is OpenAI embedding obtained

There is OpenAI embedding API https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input ...
Salty Gold Fish's user avatar
0 votes
1 answer
344 views

Word embedding and Euclidean distance

Does a transformation exist that allows to use of the Euclidean distance with the word embeddings? The Cosine distance could be a problem in my case. For example, what if I translate the vector to a ...
ozw1z5rd's user avatar
  • 171
1 vote
1 answer
180 views

How does training word embeddings bring similar words closer together?

How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
Glue's user avatar
  • 485
1 vote
2 answers
95 views

Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?

In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
Glue's user avatar
  • 485
0 votes
1 answer
61 views

Using embeddings to anonymize information

This might be a stupid question, so bear with me. I was wondering if embeddings can be used to anonymize input text. I couldn't find any information online that says that embeddings can be 1:1 decoded ...
funerr's user avatar
  • 103
1 vote
0 answers
54 views

How does softmax work for vectors?

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
Ruediger's user avatar
2 votes
1 answer
43 views

Composed cosine similarity

I have the following problem. I have 3 vectors $u,v,w$ of n dimensions. I'm able to find cosine similarities between $u$ and $v$, and between $v$ and $w$: $cosine(u,v)$ and $cosine(v,w)$. Can i use ...
Chg's user avatar
  • 21
6 votes
1 answer
1k views

How to compare the semantic similarity of text generated by large language models (GPT-3, BLOOM etc) to reference text?

Large language models, such as GPT-3, BLOOM etc, can generate open-ended text. Say I want to prompt these models to answer a question. How can I compare the semantic similarity of the answer it ...
curious-bert's user avatar
1 vote
0 answers
57 views

High loss while training word embedding

I'm trying to train (Skip-Gram) word embeddings on a dataset derived from a small piece of data (500 lines ~2500 words ~840 unique words). I'm think I'm aware that this is too small of a dataset to ...
linkedin's user avatar
0 votes
1 answer
32 views

Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)

I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
Felipe's user avatar
  • 1,014
1 vote
0 answers
45 views

Is there an analogous calculation for weighted max and min, similar to weighted average, for text embeddings?

Say I have created some numerical embedding for each word in a text corpora -- that is, for each word I have some n-dimensional vector representing the word. Now say I would like to perform some ...
Josh's user avatar
  • 308
1 vote
1 answer
976 views

Do BERT word embeddings change after training, depending on context?

Before answering "yes, of course", let me clarify what I mean: After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
Daniel von Eschwege's user avatar
1 vote
0 answers
213 views

Equivalence between one-dimensional embedding and one-hot encoding?

This may not sound like a practical question, but I am trying to create essentially a toy problem that can help me understand and apply some explainability algorithms (eg Integrated Gradients). To do ...
thecity2's user avatar
  • 1,955
4 votes
1 answer
3k views

Training Transformers: self attention weights vs embedding layer

I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
Sami Wood's user avatar
  • 143
2 votes
1 answer
339 views

Approaches for semi-supervised fine-tuning after self-supervised pre-training

My understanding is that self-supervised learning approaches approximately work like the following (I have Wav2Vec 2 in my mind here, used in speech recognition, but NLP transformer models are similar)...
phipsgabler's user avatar
0 votes
1 answer
104 views

How to encode an url into into a 60-bit 01 string? (word2vec) [closed]

At the same time, we counted the frequency of occurrences of characters in all URLs in the dataset and selected the first 59 characters with the highest frequency as valid characters. It contains 26 ...
DigitalGreyHat's user avatar
2 votes
0 answers
730 views

Alternatives to the Silhouette score in cluster analysis?

The silhouette score in my case gives quite misleading results, any alternatives? My data is a result of embedding words, which belong to one of the 20+ classes. I want to measure the "...
oliver.c's user avatar
  • 185
0 votes
1 answer
313 views

Why should the weight matrix encode word embeddings in CBOW/skip-gram?

Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings. I'm looking for an intuitive ...
Michael Ray's user avatar
4 votes
2 answers
2k views

What exactly is meant by isotropic and anisotropic with word vectors

From this paper https://aclanthology.org/D19-1006.pdf "How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings" When they say word ...
jacqui_suis's user avatar
1 vote
0 answers
204 views

Skip gram model and negative log loss likelihood

I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
MrMercury's user avatar
5 votes
2 answers
604 views

Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding

In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on). This way, each cell contains semantic and ...
Emil's user avatar
  • 361
1 vote
0 answers
37 views

Why do we call word embeddings "vectors" ? Shouldn't we call them word coordinates?

Word embeddings are often referred to as vectors. For instance, "Word embedding models are dense vector representations of words learned from a corpus of natural language" (Rosado, 2020). ...
GuillaumeL's user avatar
1 vote
1 answer
28 views

Algorithm to find closest document containing a set of strings, or variations of them

I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers. I have another dataset (B) which contains highly ...
KOB's user avatar
  • 465
0 votes
1 answer
65 views

Ways to detect noise in multi-class classification training data using text embeddings (BERT)

So I have a dataset with a column of text and and labels (5 different labels) associated with it. The labels describe the potential answer to the type of question being asked in the text column. For ...
user11715878's user avatar
1 vote
0 answers
724 views

Cosine Similarity of the word embeddings after UMAP dimensionality reduction

I want to calculate similarities of the word embeddings. As a basis I took SpaCy german corpus: nlp = spacy.load("de_core_news_lg"). I have approximately ...
Daniil Yefimov's user avatar
1 vote
0 answers
64 views

Why does latent dirichlet allocation (LDA) fail when dealing with large and heavy-tailed vocabularies?

I'm reading the 2019 paper Topic Modeling in Embedding Spaces which claims that the embedded topic model improves on these limitations of LDA. But why does LDA have these limitations—why does it fail ...
seanmachinelearning's user avatar
1 vote
1 answer
2k views

Interconnections between embeddings layer and LSTM layer

I'm trying to build a text classifier with keras using word embeddings (glove) and a RNN (in this case a LSTM) using keras. I searched in several sites and decided to start with this configuration: <...
juanpablot's user avatar
2 votes
1 answer
57 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
Ramiro Hum-Sah's user avatar
2 votes
1 answer
599 views

Should I use cosine or dot similarity inside word2vec's neural network?

I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
oliver.c's user avatar
  • 185
4 votes
1 answer
900 views

How is an embedding space optimized with respect to the loss function?

I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in ...
mesllo's user avatar
  • 679
1 vote
1 answer
317 views

Feedforward Neural Net Language Model - computational complexity (word2vec)

While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM): It consists of input, projection, hidden and output layers. At the ...
MDescamps's user avatar
0 votes
0 answers
1k views

Using BERT Embeddings + Standard ML for text classification

I am trying to automatically detect whether a text is written by a Machine or a Human. My first approach was using a TF-IDF to build features for a logistic regression classifier, where I got an ...
embedded_dev's user avatar
1 vote
0 answers
25 views

What is a word embedding approach that would work for these pre-labeled documents?

My Situation: I should start off with my end goal: I want to get a distance metric between each document and all of the other documents To get there, I first need to encode these topic labels so that ...
Jacob Myer's user avatar

1
2 3 4 5 6