Questions tagged [embeddings]
The embeddings tag has no usage guidance.
84
questions
0
votes
0
answers
9
views
How can I get knowledge graph embedding by using TransE? [duplicate]
I saw nobody answered Ssong, however now I'm struggling with the same question. Can anyone help? Thanks.
0
votes
0
answers
19
views
Can LSTMs/RNNs' Final Hidden States Serve as Effective Sequence Embeddings for Document or Sentence Analysis?
Can the final hidden state of LSTMs/RNNs be utilized as a sequence embedding for documents or sentences?
These embeddings could then be applied to tasks like classification or clustering to leverage ...
- 13
0
votes
0
answers
33
views
How is OpenAI embedding obtained
There is OpenAI embedding API
https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input ...
- 459
0
votes
1
answer
35
views
Why the positional embeddings for different positions are not confined
for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))
we know <...
- 169
0
votes
1
answer
55
views
Why the positional embeddings for a specific position i, for each embedding element is different
for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))
we know <...
- 169
1
vote
0
answers
23
views
Best way to encode a variable number of points in a 2D space as features
I am presented with this problem: given a set of points (the number can vary, and their identity is not fixed across observations) distributed in a bounded 2D space (say $x \in [0,1]$, $y \in [0,1]$), ...
- 435
1
vote
1
answer
226
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
0
votes
0
answers
28
views
How to do column weighted truncated SVD?
I have an unusual case where I need to combine two vector spaces but weight one more than the other. Rather than discussing my specific use case, it's likely easier to imagine we trained two word2vec ...
- 298
1
vote
0
answers
24
views
Dimensionality reduction that preserves non-trivial similarity
I have a set of vectors $x_i \in\mathbb{R}^m$ and a similarity function f that quantifies how similar $x_i,x_j$ are. Unfortunately, calculating f takes a lot of time.
I want to use some neural ...
- 21
0
votes
0
answers
53
views
BERT embeddings are different in batches, compared to single input
I'm using BERT embeddings in an SVM. For computational efficiency, I'm batching my data, feeding them into BERT to get the embeddings from the last 4 hidden layers, and then using these embeddings in ...
- 31
1
vote
0
answers
92
views
Finding a latent representation of a high-cardinality one-hot encoded variable [duplicate]
I am working on a clustering project on a dataset that has some numerical variables, and one categorical variable with very high cardinality (~200 values). I was thinking if it is possible to create ...
- 11
0
votes
0
answers
20
views
Which pre-trained model to use to generate embeddings from shop names?
Good afternoon!
I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
- 93
1
vote
1
answer
517
views
Do BERT word embeddings change after training, depending on context?
Before answering "yes, of course", let me clarify what I mean:
After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
0
votes
0
answers
91
views
Cycle detection on unsupervised time series data
i have some video data of production lines of some manufactories. In every video, an operator does the same 3-4 steps periodically for the entire video. Each periods of same steps is called cycle and ...
- 1
0
votes
0
answers
30
views
Projecting data into CCA components
When I perform CCA and get the projected features. The eigenvectors for the Y data. Suppose I want to project new data into it? Do I do $NewData*Y_{compenents}$ or $Y_{componenets}*NewData$?
- 368
1
vote
1
answer
21
views
Algorithm to find closest document containing a set of strings, or variations of them
I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers.
I have another dataset (B) which contains highly ...
- 385
1
vote
0
answers
31
views
Check if the embeddings preserve the true space structure
I obtained the embeddings from the pre-classification layer of a neural network for 600K instances. Each embedding's dimension is 1024, thus we have a sample of the embedding space, $\mathcal{X} \...
- 236
2
votes
1
answer
51
views
Text similarity for badly written text
Consider the following scenario:
Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
- 287
4
votes
1
answer
607
views
How is an embedding space optimized with respect to the loss function?
I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in ...
- 679
1
vote
0
answers
56
views
Information preserved in the kernel mean embedding
I have recently been introduced to the kernel mean embedding of distributions, that is the map
$$\mu: \mathcal{M}^{1}_{+}(X) \rightarrow \mathcal{H} \\ \mu(P) := \int \phi(x) dP(x)$$
where $K$ is a ...
- 298
0
votes
0
answers
140
views
How to improve the PMI (Pointwise Mutual Information) Quality for document based PMI
Generating word embeddings from the PMI is well understood and known to be equivalent to SGNS (skipgram negative-sampling) under certain conditions. I was able to get good quality word embedding using ...
yangsuli
0
votes
0
answers
187
views
How to optimize K-means to eliminate outliers and unrelated clusters?
I clustered document embeddings with K-Means. Embeddings have 2048 dimensions. Now, i am trying to optimize clustering. There are two problems. 1- Some clusters may have outlier samples. 2- Sometimes,...
- 1
1
vote
0
answers
166
views
Model that gracefully handles missing values at inference
Say I have a neural network that predicts whether a restaurant will be successful in a given neighborhood and uses the following features:
type of food served
type of music played
type of decoration
...
- 571
0
votes
0
answers
222
views
How to train a custom embedding?
I have data with a lot of categorical features. The cardinality of some of these features is quite big (>100), so I want to avoid using one-hot encoding. The idea is to use an embedding.
The ...
- 343
1
vote
0
answers
47
views
Create score from various features/variables
I have 4 measurements for different sensors and I would like to somehow create a score for each measurement ranging from 1-10. I have tried performing PCA with 1 component on these 4 columns and then ...
- 159
0
votes
0
answers
53
views
Using compositional data analysis to represent chemical compounds
I've recently got some insights about compositional data analysis, wondering whether it could be suitable for the framework I'm currently in.
Recently, I've been very interested in trying to find some ...
- 557
1
vote
0
answers
72
views
Definition of the word "embedding"
The mathematical definition of the word "embedding" requires the mapping to be injective, so in that context one speaks of, for example, embedding real numbers in complex numbers (ie, ...
- 916
5
votes
3
answers
2k
views
Embedding data into a larger dimension space
Embeddings or latent spaces are vector spaces that we embed our initial data into that for further processing. The benefit of doing so as far as I am aware, is to reduce the dimension. Often data has ...
- 207
9
votes
1
answer
965
views
If the curse of dimensionality exists, how does embedding search work?
The curse of dimensionality tells us if the dimension is high, the distance metric will stop working, i.e., everyone will be close to everyone.
However, many machine learning retrieval systems rely on ...
- 35.8k
0
votes
0
answers
28
views
Word-embedding does not provide expected relations between words
I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
1
vote
1
answer
41
views
Generating embeddings for languages without a written representation?
I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as ...
- 11.5k
3
votes
0
answers
111
views
Is there an MDS/embedding algorithm that is more suitable to the goal of clustering a graph
I am testing ideas on clustering a particular graph. After testing a set of graph clustering/community detection algorithms I thought about mapping the graph to a vector space and using vector space ...
- 5,357
4
votes
1
answer
162
views
ArcFace - How to compute $\cos(t+m)$ when $t+m > \pi$
I am trying to understand the ArcFace Implementation and I am stuck at one condition.
If the $ \cos(t) > \cos(\pi -m)$ then $t + m > \pi$. In this case the way how we're computing $\cos(t+m)$ is ...
- 141
1
vote
0
answers
143
views
Learn sentence embeddings from a sequence of token embeddings
I want to build a sentence classifier that takes the sentence as a sequence of token embeddings. I'm specifically interested in the methodology for learning the sentence embedding from the sequence of ...
- 111
1
vote
1
answer
244
views
Extracting word embedding features of a sentence using Transformer-XL
As you know, there are several pre-trained models that we can use to extract word embeddings.
As an example, I can use the following codes to retrieve word2vec features of my text:
...
- 395
1
vote
0
answers
506
views
How to pass an array of variable length to the input of the neural network?
I have a bunch of two-dimensional points that I want to feed into my neural network as an input. Those points are positions of the visible obstacles around my agent. The main challenge is that the ...
- 111
1
vote
0
answers
126
views
Embeddings model: out of sample prediction with Keras for Collaborative Filtering
I have been trying to play with an example on Collaborative Filtering for Movie Recommendations (keras.io), which builds embedding layers for movies and users.
Now, in a regular pre-trained word- and ...
- 11
1
vote
1
answer
388
views
What's the best practice for dealing with OOV characters?
I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
- 23
1
vote
1
answer
278
views
What is the relationship between embeddings and deep metric learning?
Embeddings and deep metric learning seem architecturally identical. They both rely on using some hidden layer's vector representation of an input.
What is the difference between the two? Are ...
- 1,011
6
votes
1
answer
3k
views
What is embedding? (in the context of dimensionality reduction)
In the context of dimensionality reduction one often uses word embedding, which seems to me a rather technical mathematical term, which rather stands out compared to the rest of the discussion, which ...
- 3,512
1
vote
0
answers
2k
views
Which dimensionality reduction technique works well for BERT sentence embeddings?
I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
- 11
2
votes
0
answers
325
views
dimension of input layer for embeddings in Keras
It is not clear to me whether there is any difference between specifying the input dimension
Input(shape=(20,)) or not ...
- 31
0
votes
0
answers
78
views
Is there an algorithm for placing 2-dimensional embeddings into a grid so they can be displayed?
I’m using PCA to reduce images down to 2d embeddings and I’d like to display the images in a grid.
The Pudding did something like this with book covers, using tsne and a library called RasterFairy, by ...
- 113
1
vote
0
answers
266
views
Why are we interested in gradient with respect to input?
I am learning about sampling methods for Deep Embedding Learning. I was reading an article named: "Sampling Matters in Deep Embedding Learning" (https://arxiv.org/abs/1706.07567).
In the ...
- 11
3
votes
1
answer
1k
views
What is the difference between embedding in pure math and embedding in ML?
In ML the term "embedding" gets tossed around a lot and the term basically means the construction of a function that takes a high-dimensional vector to a low-dimensional vector in such a way ...
1
vote
2
answers
94
views
Why are character level models considered less effective than word level models?
I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
0
votes
1
answer
137
views
Why do Dense layers perform better than a mix of Conv Layers, Recurrent Layers on Sentiment Analysis with BERT emebddings?
I have used BERT to make embeddings out of the imdb review dataset and I am trying out some models to check their perfomance on sentiment analysis (0 for the bad reviews and 1 for the good ones). I ...
- 151
2
votes
1
answer
211
views
Is there any paper about applications of Deep Metric Learning on regression problem?
I'm trying to solve a problem in the field of transfer learning, more specifically, domain adaption where both the source domain and target domain are labeled. Basically it's to predict the ...
- 133
4
votes
1
answer
2k
views
Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?
I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine):
Note that (as is usually the case in Machine Learning) we ...
- 3,510
1
vote
0
answers
125
views
Adam converges while SGD does not improve at all
I am trying to build a model based movie recommendation system with a neural network.
The architecture looks as follows:
...
- 111