Skip to main content

Questions tagged [embeddings]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Can 3D convolutions appropriately capture a frozen embedding space?

My project is a strange combination of NLP and Computer Vision. I have datapoints of 3D tensor where each element is a token in an NLP vocabulary. The vocabulary is around 1000 unique "words"...
schmixi's user avatar
  • 43
0 votes
0 answers
10 views

How can I get knowledge graph embedding by using TransE? [duplicate]

I saw nobody answered Ssong, however now I'm struggling with the same question. Can anyone help? Thanks.
Yolanda Zhu's user avatar
0 votes
0 answers
26 views

Can LSTMs/RNNs' Final Hidden States Serve as Effective Sequence Embeddings for Document or Sentence Analysis?

Can the final hidden state of LSTMs/RNNs be utilized as a sequence embedding for documents or sentences? These embeddings could then be applied to tasks like classification or clustering to leverage ...
palash behra's user avatar
0 votes
0 answers
42 views

How is OpenAI embedding obtained

There is OpenAI embedding API https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input ...
Salty Gold Fish's user avatar
0 votes
1 answer
170 views

Why the positional embeddings for different positions are not confined

for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension))) we know <...
Farhang Amaji's user avatar
0 votes
1 answer
178 views

Why the positional embeddings for a specific position i, for each embedding element is different

for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension))) we know <...
Farhang Amaji's user avatar
1 vote
0 answers
37 views

Best way to encode a variable number of points in a 2D space as features

I am presented with this problem: given a set of points (the number can vary, and their identity is not fixed across observations) distributed in a bounded 2D space (say $x \in [0,1]$, $y \in [0,1]$), ...
gianMa's user avatar
  • 435
1 vote
1 answer
823 views

How do I go from embeddings to queries, keys and values in the Transformer model?

I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
Abysmal_query's user avatar
1 vote
0 answers
24 views

Dimensionality reduction that preserves non-trivial similarity

I have a set of vectors $x_i \in\mathbb{R}^m$ and a similarity function f that quantifies how similar $x_i,x_j$ are. Unfortunately, calculating f takes a lot of time. I want to use some neural ...
username_94's user avatar
1 vote
0 answers
127 views

Finding a latent representation of a high-cardinality one-hot encoded variable [duplicate]

I am working on a clustering project on a dataset that has some numerical variables, and one categorical variable with very high cardinality (~200 values). I was thinking if it is possible to create ...
ockham_blade's user avatar
1 vote
1 answer
981 views

Do BERT word embeddings change after training, depending on context?

Before answering "yes, of course", let me clarify what I mean: After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
Daniel von Eschwege's user avatar
1 vote
1 answer
28 views

Algorithm to find closest document containing a set of strings, or variations of them

I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers. I have another dataset (B) which contains highly ...
KOB's user avatar
  • 465
1 vote
0 answers
35 views

Check if the embeddings preserve the true space structure

I obtained the embeddings from the pre-classification layer of a neural network for 600K instances. Each embedding's dimension is 1024, thus we have a sample of the embedding space, $\mathcal{X} \...
rando's user avatar
  • 308
2 votes
1 answer
57 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
Ramiro Hum-Sah's user avatar
4 votes
1 answer
904 views

How is an embedding space optimized with respect to the loss function?

I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in ...
mesllo's user avatar
  • 679
1 vote
0 answers
66 views

Information preserved in the kernel mean embedding

I have recently been introduced to the kernel mean embedding of distributions, that is the map $$\mu: \mathcal{M}^{1}_{+}(X) \rightarrow \mathcal{H} \\ \mu(P) := \int \phi(x) dP(x)$$ where $K$ is a ...
Claudio Moneo's user avatar
0 votes
0 answers
155 views

How to improve the PMI (Pointwise Mutual Information) Quality for document based PMI

Generating word embeddings from the PMI is well understood and known to be equivalent to SGNS (skipgram negative-sampling) under certain conditions. I was able to get good quality word embedding using ...
user avatar
0 votes
0 answers
225 views

How to optimize K-means to eliminate outliers and unrelated clusters?

I clustered document embeddings with K-Means. Embeddings have 2048 dimensions. Now, i am trying to optimize clustering. There are two problems. 1- Some clusters may have outlier samples. 2- Sometimes,...
Alper M.'s user avatar
1 vote
0 answers
279 views

Model that gracefully handles missing values at inference

Say I have a neural network that predicts whether a restaurant will be successful in a given neighborhood and uses the following features: type of food served type of music played type of decoration ...
Jivan's user avatar
  • 571
0 votes
0 answers
316 views

How to train a custom embedding?

I have data with a lot of categorical features. The cardinality of some of these features is quite big (>100), so I want to avoid using one-hot encoding. The idea is to use an embedding. The ...
DocDriven's user avatar
  • 343
1 vote
0 answers
63 views

Create score from various features/variables

I have 4 measurements for different sensors and I would like to somehow create a score for each measurement ranging from 1-10. I have tried performing PCA with 1 component on these 4 columns and then ...
azal's user avatar
  • 159
1 vote
0 answers
65 views

Using compositional data analysis to represent chemical compounds

I've recently got some insights about compositional data analysis, wondering whether it could be suitable for the framework I'm currently in. Recently, I've been very interested in trying to find some ...
James Arten's user avatar
1 vote
0 answers
88 views

Definition of the word "embedding"

The mathematical definition of the word "embedding" requires the mapping to be injective, so in that context one speaks of, for example, embedding real numbers in complex numbers (ie, ...
Lei Huang's user avatar
  • 966
6 votes
3 answers
3k views

Embedding data into a larger dimension space

Embeddings or latent spaces are vector spaces that we embed our initial data into that for further processing. The benefit of doing so as far as I am aware, is to reduce the dimension. Often data has ...
user127776's user avatar
15 votes
2 answers
1k views

If the curse of dimensionality exists, how does embedding search work?

The curse of dimensionality tells us if the dimension is high, the distance metric will stop working, i.e., everyone will be close to everyone. However, many machine learning retrieval systems rely on ...
Haitao Du's user avatar
  • 37.2k
0 votes
0 answers
29 views

Word-embedding does not provide expected relations between words

I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
user320342's user avatar
5 votes
1 answer
48 views

Generating embeddings for languages without a written representation?

I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as ...
Skander H.'s user avatar
  • 12.1k
3 votes
0 answers
149 views

Is there an MDS/embedding algorithm that is more suitable to the goal of clustering a graph

I am testing ideas on clustering a particular graph. After testing a set of graph clustering/community detection algorithms I thought about mapping the graph to a vector space and using vector space ...
Jacques Wainer's user avatar
4 votes
1 answer
191 views

ArcFace - How to compute $\cos(t+m)$ when $t+m > \pi$

I am trying to understand the ArcFace Implementation and I am stuck at one condition. If the $ \cos(t) > \cos(\pi -m)$ then $t + m > \pi$. In this case the way how we're computing $\cos(t+m)$ is ...
pawols's user avatar
  • 141
1 vote
0 answers
166 views

Learn sentence embeddings from a sequence of token embeddings

I want to build a sentence classifier that takes the sentence as a sequence of token embeddings. I'm specifically interested in the methodology for learning the sentence embedding from the sequence of ...
D Hudson's user avatar
  • 111
1 vote
1 answer
278 views

Extracting word embedding features of a sentence using Transformer-XL

As you know, there are several pre-trained models that we can use to extract word embeddings. As an example, I can use the following codes to retrieve word2vec features of my text: ...
Kadaj13's user avatar
  • 395
1 vote
0 answers
563 views

How to pass an array of variable length to the input of the neural network?

I have a bunch of two-dimensional points that I want to feed into my neural network as an input. Those points are positions of the visible obstacles around my agent. The main challenge is that the ...
George's user avatar
  • 111
1 vote
0 answers
139 views

Embeddings model: out of sample prediction with Keras for Collaborative Filtering

I have been trying to play with an example on Collaborative Filtering for Movie Recommendations (keras.io), which builds embedding layers for movies and users. Now, in a regular pre-trained word- and ...
vvseva's user avatar
  • 11
2 votes
1 answer
481 views

What's the best practice for dealing with OOV characters?

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
gmedina-v's user avatar
1 vote
1 answer
366 views

What is the relationship between embeddings and deep metric learning?

Embeddings and deep metric learning seem architecturally identical. They both rely on using some hidden layer's vector representation of an input. What is the difference between the two? Are ...
kennysong's user avatar
  • 1,061
6 votes
1 answer
4k views

What is embedding? (in the context of dimensionality reduction)

In the context of dimensionality reduction one often uses word embedding, which seems to me a rather technical mathematical term, which rather stands out compared to the rest of the discussion, which ...
Roger V.'s user avatar
  • 4,429
1 vote
0 answers
2k views

Which dimensionality reduction technique works well for BERT sentence embeddings?

I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
Selina 's user avatar
2 votes
0 answers
338 views

dimension of input layer for embeddings in Keras

It is not clear to me whether there is any difference between specifying the input dimension Input(shape=(20,)) or not ...
learner's user avatar
  • 31
0 votes
0 answers
88 views

Is there an algorithm for placing 2-dimensional embeddings into a grid so they can be displayed?

I’m using PCA to reduce images down to 2d embeddings and I’d like to display the images in a grid. The Pudding did something like this with book covers, using tsne and a library called RasterFairy, by ...
A Poor's user avatar
  • 113
1 vote
0 answers
282 views

Why are we interested in gradient with respect to input?

I am learning about sampling methods for Deep Embedding Learning. I was reading an article named: "Sampling Matters in Deep Embedding Learning" (https://arxiv.org/abs/1706.07567). In the ...
Mike's user avatar
  • 11
4 votes
1 answer
2k views

What is the difference between embedding in pure math and embedding in ML?

In ML the term "embedding" gets tossed around a lot and the term basically means the construction of a function that takes a high-dimensional vector to a low-dimensional vector in such a way ...
Curaçao Hajek's user avatar
1 vote
2 answers
120 views

Why are character level models considered less effective than word level models?

I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
yash karbhari's user avatar
0 votes
1 answer
168 views

Why do Dense layers perform better than a mix of Conv Layers, Recurrent Layers on Sentiment Analysis with BERT emebddings?

I have used BERT to make embeddings out of the imdb review dataset and I am trying out some models to check their perfomance on sentiment analysis (0 for the bad reviews and 1 for the good ones). I ...
Borja_042's user avatar
  • 151
2 votes
1 answer
237 views

Is there any paper about applications of Deep Metric Learning on regression problem?

I'm trying to solve a problem in the field of transfer learning, more specifically, domain adaption where both the source domain and target domain are labeled. Basically it's to predict the ...
PokeLu's user avatar
  • 133
4 votes
2 answers
2k views

Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?

I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine): Note that (as is usually the case in Machine Learning) we ...
stackoverflowuser2010's user avatar
1 vote
0 answers
167 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
Simon's user avatar
  • 111
11 votes
2 answers
3k views

Why BERT use learned positional embedding?

Compared with sinusoidal positional encoding used in Transformer, BERT's learned-lookup-table solution has 2 drawbacks in my mind: Fixed length Cannot reflect relative distance Could anyone please ...
eric2323223's user avatar
3 votes
1 answer
1k views

What are state of the art methods for creating embeddings for sets?

I want to create embeddings in $R^D$ for sets. So I want a function (probably a neural network) that takes in a set $ S = \{ s_1, \dots, s_n \} $ (and ideally of any size, so the number of elements ...
Charlie Parker's user avatar
1 vote
3 answers
66 views

General mathematical definition of a score

I understand what scores are in PCA, in particular this answer gives a good mathematical formulation: (Scores) are projections of the centred data in the linear space defined by the eigenvectors. ...
kamilazdybal's user avatar
0 votes
1 answer
593 views

Embedded markov chain example

I have an example of in my textbook of an "embeded markov chain", where I don't understand one step. Suppose that $(X_n)_{n\geq 0}$ is Markov$(\lambda, P)$. $\lambda$ is the initial distribution and ...
user avatar