Questions tagged [embeddings]
The embeddings tag has no usage guidance.
84 questions
0
votes
0
answers
15
views
Embeddings in time series prediction
Increasingly, I’ve noted that embeddings are used in pure prediction ML tasks. For example, instead of predicting whether user i will purchase item i and thereby adding thousands or millions of inputs ...
- 3,520
0
votes
0
answers
23
views
Words’ similarity formula based on the context words
I am working on my word embedding calculation algorithm and stuck with a similarity formula.
I assume that this could be easily derived formally with statistics and probabilities, but I fail to do so. ...
- 203
0
votes
0
answers
25
views
Can you use any hidden layer in a neural network as embedding?
Embeddings from encoder comes from hidden layer of the neural work, but there are many hidden layer inside a particular neural network
How do architectures like Asymmetric Dual Encoder decide which ...
- 11
1
vote
1
answer
156
views
How does two tower model map to shared embedding space for two different type of entity?
A canonical example is say you have user and merchandise
user (feature: age, location....)
merchandise (feature: type, size, .....)
And you want to create embedding to map user and merchandise to same ...
- 11
3
votes
0
answers
34
views
Convergence of kernel mean embeddings
Let $k(\cdot,\cdot)$ be a bounded kernel and $\mathcal{H}$ its associated RKHS. Define the kernel mean embedding $\mu=\int k(\cdot,x) \, dP_X(x)$ and let $\hat{\mu}=\frac{1}{n}\sum k(x_i,\cdot)$ be ...
- 122
0
votes
0
answers
19
views
Shallow better than wide deep learning embedding models?
I'm training a two-tower recommender embedding model where one tower represents users and another represents items. User and item embeddings should be close when a user clicked an item and far part ...
- 6,767
0
votes
0
answers
21
views
Can 3D convolutions appropriately capture a frozen embedding space?
My project is a strange combination of NLP and Computer Vision.
I have datapoints of 3D tensor where each element is a token in an NLP vocabulary. The vocabulary is around 1000 unique "words"...
- 43
0
votes
0
answers
53
views
How is OpenAI embedding obtained
There is OpenAI embedding API
https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input ...
- 519
0
votes
1
answer
226
views
Why the positional embeddings for different positions are not confined
for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))
we know <...
- 169
0
votes
1
answer
208
views
Why the positional embeddings for a specific position i, for each embedding element is different
for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))
we know <...
- 169
1
vote
0
answers
48
views
Best way to encode a variable number of points in a 2D space as features
I am presented with this problem: given a set of points (the number can vary, and their identity is not fixed across observations) distributed in a bounded 2D space (say $x \in [0,1]$, $y \in [0,1]$), ...
- 445
1
vote
1
answer
1k
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
1
vote
0
answers
24
views
Dimensionality reduction that preserves non-trivial similarity
I have a set of vectors $x_i \in\mathbb{R}^m$ and a similarity function f that quantifies how similar $x_i,x_j$ are. Unfortunately, calculating f takes a lot of time.
I want to use some neural ...
- 21
1
vote
0
answers
137
views
Finding a latent representation of a high-cardinality one-hot encoded variable [duplicate]
I am working on a clustering project on a dataset that has some numerical variables, and one categorical variable with very high cardinality (~200 values). I was thinking if it is possible to create ...
- 11
1
vote
1
answer
1k
views
Do BERT word embeddings change after training, depending on context?
Before answering "yes, of course", let me clarify what I mean:
After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
1
vote
1
answer
28
views
Algorithm to find closest document containing a set of strings, or variations of them
I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers.
I have another dataset (B) which contains highly ...
- 465
1
vote
0
answers
35
views
Check if the embeddings preserve the true space structure
I obtained the embeddings from the pre-classification layer of a neural network for 600K instances. Each embedding's dimension is 1024, thus we have a sample of the embedding space, $\mathcal{X} \...
- 328
2
votes
1
answer
58
views
Text similarity for badly written text
Consider the following scenario:
Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
- 337
4
votes
1
answer
994
views
How is an embedding space optimized with respect to the loss function?
I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in ...
- 679
1
vote
0
answers
77
views
Information preserved in the kernel mean embedding
I have recently been introduced to the kernel mean embedding of distributions, that is the map
$$\mu: \mathcal{M}^{1}_{+}(X) \rightarrow \mathcal{H} \\ \mu(P) := \int \phi(x) dP(x)$$
where $K$ is a ...
- 318
0
votes
0
answers
161
views
How to improve the PMI (Pointwise Mutual Information) Quality for document based PMI
Generating word embeddings from the PMI is well understood and known to be equivalent to SGNS (skipgram negative-sampling) under certain conditions. I was able to get good quality word embedding using ...
yangsuli
0
votes
0
answers
238
views
How to optimize K-means to eliminate outliers and unrelated clusters?
I clustered document embeddings with K-Means. Embeddings have 2048 dimensions. Now, i am trying to optimize clustering. There are two problems. 1- Some clusters may have outlier samples. 2- Sometimes,...
- 1
1
vote
0
answers
296
views
Model that gracefully handles missing values at inference
Say I have a neural network that predicts whether a restaurant will be successful in a given neighborhood and uses the following features:
type of food served
type of music played
type of decoration
...
- 571
0
votes
0
answers
338
views
How to train a custom embedding?
I have data with a lot of categorical features. The cardinality of some of these features is quite big (>100), so I want to avoid using one-hot encoding. The idea is to use an embedding.
The ...
- 343
1
vote
0
answers
69
views
Create score from various features/variables
I have 4 measurements for different sensors and I would like to somehow create a score for each measurement ranging from 1-10. I have tried performing PCA with 1 component on these 4 columns and then ...
- 159
1
vote
0
answers
67
views
Using compositional data analysis to represent chemical compounds
I've recently got some insights about compositional data analysis, wondering whether it could be suitable for the framework I'm currently in.
Recently, I've been very interested in trying to find some ...
- 659
1
vote
0
answers
91
views
Definition of the word "embedding"
The mathematical definition of the word "embedding" requires the mapping to be injective, so in that context one speaks of, for example, embedding real numbers in complex numbers (ie, ...
- 976
6
votes
2
answers
3k
views
Embedding data into a larger dimension space
Embeddings or latent spaces are vector spaces that we embed our initial data into that for further processing. The benefit of doing so as far as I am aware, is to reduce the dimension. Often data has ...
- 249
15
votes
2
answers
2k
views
If the curse of dimensionality exists, how does embedding search work?
The curse of dimensionality tells us if the dimension is high, the distance metric will stop working, i.e., everyone will be close to everyone.
However, many machine learning retrieval systems rely on ...
- 37.3k
0
votes
0
answers
29
views
Word-embedding does not provide expected relations between words
I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
5
votes
1
answer
50
views
Generating embeddings for languages without a written representation?
I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as ...
- 12.1k
3
votes
0
answers
156
views
Is there an MDS/embedding algorithm that is more suitable to the goal of clustering a graph
I am testing ideas on clustering a particular graph. After testing a set of graph clustering/community detection algorithms I thought about mapping the graph to a vector space and using vector space ...
- 5,452
4
votes
1
answer
197
views
ArcFace - How to compute $\cos(t+m)$ when $t+m > \pi$
I am trying to understand the ArcFace Implementation and I am stuck at one condition.
If the $ \cos(t) > \cos(\pi -m)$ then $t + m > \pi$. In this case the way how we're computing $\cos(t+m)$ is ...
- 141
1
vote
0
answers
174
views
Learn sentence embeddings from a sequence of token embeddings
I want to build a sentence classifier that takes the sentence as a sequence of token embeddings. I'm specifically interested in the methodology for learning the sentence embedding from the sequence of ...
- 111
1
vote
1
answer
279
views
Extracting word embedding features of a sentence using Transformer-XL
As you know, there are several pre-trained models that we can use to extract word embeddings.
As an example, I can use the following codes to retrieve word2vec features of my text:
...
- 395
1
vote
0
answers
575
views
How to pass an array of variable length to the input of the neural network?
I have a bunch of two-dimensional points that I want to feed into my neural network as an input. Those points are positions of the visible obstacles around my agent. The main challenge is that the ...
- 111
1
vote
0
answers
141
views
Embeddings model: out of sample prediction with Keras for Collaborative Filtering
I have been trying to play with an example on Collaborative Filtering for Movie Recommendations (keras.io), which builds embedding layers for movies and users.
Now, in a regular pre-trained word- and ...
- 11
2
votes
1
answer
515
views
What's the best practice for dealing with OOV characters?
I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
- 33
2
votes
1
answer
391
views
What is the relationship between embeddings and deep metric learning?
Embeddings and deep metric learning seem architecturally identical. They both rely on using some hidden layer's vector representation of an input.
What is the difference between the two? Are ...
- 1,081
6
votes
1
answer
4k
views
What is embedding? (in the context of dimensionality reduction)
In the context of dimensionality reduction one often uses word embedding, which seems to me a rather technical mathematical term, which rather stands out compared to the rest of the discussion, which ...
- 4,487
1
vote
0
answers
2k
views
Which dimensionality reduction technique works well for BERT sentence embeddings?
I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
- 11
2
votes
0
answers
341
views
dimension of input layer for embeddings in Keras
It is not clear to me whether there is any difference between specifying the input dimension
Input(shape=(20,)) or not ...
- 31
0
votes
0
answers
88
views
Is there an algorithm for placing 2-dimensional embeddings into a grid so they can be displayed?
I’m using PCA to reduce images down to 2d embeddings and I’d like to display the images in a grid.
The Pudding did something like this with book covers, using tsne and a library called RasterFairy, by ...
- 123
1
vote
0
answers
285
views
Why are we interested in gradient with respect to input?
I am learning about sampling methods for Deep Embedding Learning. I was reading an article named: "Sampling Matters in Deep Embedding Learning" (https://arxiv.org/abs/1706.07567).
In the ...
- 11
4
votes
1
answer
2k
views
What is the difference between embedding in pure math and embedding in ML?
In ML the term "embedding" gets tossed around a lot and the term basically means the construction of a function that takes a high-dimensional vector to a low-dimensional vector in such a way ...
- 129
1
vote
2
answers
127
views
Why are character level models considered less effective than word level models?
I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
0
votes
1
answer
175
views
Why do Dense layers perform better than a mix of Conv Layers, Recurrent Layers on Sentiment Analysis with BERT emebddings?
I have used BERT to make embeddings out of the imdb review dataset and I am trying out some models to check their perfomance on sentiment analysis (0 for the bad reviews and 1 for the good ones). I ...
- 151
2
votes
1
answer
254
views
Is there any paper about applications of Deep Metric Learning on regression problem?
I'm trying to solve a problem in the field of transfer learning, more specifically, domain adaption where both the source domain and target domain are labeled. Basically it's to predict the ...
- 133
4
votes
2
answers
2k
views
Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?
I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine):
Note that (as is usually the case in Machine Learning) we ...
- 3,630
1
vote
0
answers
171
views
Adam converges while SGD does not improve at all
I am trying to build a model based movie recommendation system with a neural network.
The architecture looks as follows:
...
- 111