Questions tagged [embeddings]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
21 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
user avatar
0 votes
0 answers
10 views

embeddings for representing sequences in which items also have numeric attributes/features

My data is a collection of sequences of items (or events or webpages), in which each item also has associated numeric features. An example could be the following scenario: An on-demand movie ...
user avatar
0 votes
0 answers
12 views

Updating temporal embeddings depending on the input

I'm building a forecasting model and I'm using a temporal embedding along with a positional embedding following the same architecture as Informer. ( https://arxiv.org/abs/2012.07436 ) My problem is ...
user avatar
1 vote
0 answers
11 views

How is an embedding space optimized with respect to the loss function?

I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in ...
user avatar
  • 609
1 vote
0 answers
32 views

Information preserved in the kernel mean embedding

I have recently been introduced to the kernel mean embedding of distributions, that is the map $$\mu: \mathcal{M}^{1}_{+}(X) \rightarrow \mathcal{H} \\ \mu(P) := \int \phi(x) dP(x)$$ where $K$ is a ...
user avatar
0 votes
0 answers
11 views

Word embedding to support OOV words for identity embedding

I want to create a model that performs a user-id embedding (hash of a user) for a Graph Neural Network learning task, the problem I am facing is that I might have a very large corpus of users, which ...
user avatar
0 votes
0 answers
10 views

Embedding extraction -> Classifier VS Embedding learning+ Classification on-the-fly?

I have two questions: How should we compare in general which of the following perform better? I have a graph and would like to perform a graph classification task. Is it better to extract graph ...
user avatar
0 votes
0 answers
71 views

Triplet loss for text embedding and text similarity? [duplicate]

I am working on a triplet loss based model for text embedding. Short description: I have a database about online shop, I need to find the suitble product when users enter a text on search bar. I ...
user avatar
0 votes
0 answers
48 views

How to improve the PMI (Pointwise Mutual Information) Quality for document based PMI

Generating word embeddings from the PMI is well understood and known to be equivalent to SGNS (skipgram negative-sampling) under certain conditions. I was able to get good quality word embedding using ...
user avatar
0 votes
0 answers
56 views

How to optimize K-means to eliminate outliers and unrelated clusters?

I clustered document embeddings with K-Means. Embeddings have 2048 dimensions. Now, i am trying to optimize clustering. There are two problems. 1- Some clusters may have outlier samples. 2- Sometimes,...
user avatar
1 vote
0 answers
27 views

Model that gracefully handles missing values at inference

Say I have a neural network that predicts whether a restaurant will be successful in a given neighborhood and uses the following features: type of food served type of music played type of decoration ...
user avatar
  • 509
0 votes
0 answers
44 views

Best encoding for a multiple-value categorical feature

I have a neural network classifier which works reasonably well, trying to predict city of destination given (among other things) city of origin. The city of origin is encoded into an integer form (e.g....
user avatar
  • 509
0 votes
0 answers
41 views

How to train a custom embedding?

I have data with a lot of categorical features. The cardinality of some of these features is quite big (>100), so I want to avoid using one-hot encoding. The idea is to use an embedding. The ...
user avatar
  • 283
1 vote
0 answers
12 views

Create score from various features/variables

I have 4 measurements for different sensors and I would like to somehow create a score for each measurement ranging from 1-10. I have tried performing PCA with 1 component on these 4 columns and then ...
user avatar
  • 159
0 votes
0 answers
18 views

spherical embedding of image data

Embedding positive data images onto a spherical manifold using the spherical embedding method doesn't guarantee the generation of positive directional quantities as the embedding positions are ...
user avatar
0 votes
0 answers
28 views

Using compositional data analysis to represent chemical compounds

I've recently got some insights about compositional data analysis, wondering whether it could be suitable for the framework I'm currently in. Recently, I've been very interested in trying to find some ...
user avatar
1 vote
0 answers
45 views

Definition of the word "embedding"

The mathematical definition of the word "embedding" requires the mapping to be injective, so in that context one speaks of, for example, embedding real numbers in complex numbers (ie, ...
user avatar
  • 796
0 votes
0 answers
56 views

training embedding, autoencoder and classification. Not classification learning

I have implemented a network that has an autoencoder and after that a Multiple layer perceptron (MLP). Before autoencoder I used embedding layers. The embeddings’ outputs are inputs of autoencoder. I ...
user avatar
3 votes
2 answers
438 views

Embedding data into a larger dimension space

Embeddings or latent spaces are vector spaces that we embed our initial data into that for further processing. The benefit of doing so as far as I am aware, is to reduce the dimension. Often data has ...
user avatar
0 votes
0 answers
63 views

Input embedding layer keras (CBOW)

I have a question about the treatment that the embedding layer does to the inputs in keras. model.add(Embedding(input_dim, output_dim, input_length=max_length)) In an embedding layer, for a CBOW model,...
user avatar
  • 211
5 votes
1 answer
363 views

If the curse of dimensionality exists, how does embedding search work?

The curse of dimensionality tells us if the dimension is high, the distance metric will stop working, i.e., everyone will be close to everyone. However, many machine learning retrieval systems rely on ...
user avatar
  • 33.3k
0 votes
0 answers
23 views

Word-embedding does not provide expected relations between words

I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
user avatar
1 vote
1 answer
36 views

Generating embeddings for languages without a written representation?

I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as ...
user avatar
  • 10.7k
2 votes
0 answers
58 views

Is there an MDS/embedding algorithm that is more suitable to the goal of clustering a graph

I am testing ideas on clustering a particular graph. After testing a set of graph clustering/community detection algorithms I thought about mapping the graph to a vector space and using vector space ...
user avatar
4 votes
1 answer
127 views

ArcFace - How to compute $\cos(t+m)$ when $t+m > \pi$

I am trying to understand the ArcFace Implementation and I am stuck at one condition. If the $ \cos(t) > \cos(\pi -m)$ then $t + m > \pi$. In this case the way how we're computing $\cos(t+m)$ is ...
user avatar
  • 141
1 vote
0 answers
47 views

Learn sentence embeddings from a sequence of token embeddings

I want to build a sentence classifier that takes the sentence as a sequence of token embeddings. I'm specifically interested in the methodology for learning the sentence embedding from the sequence of ...
user avatar
  • 111
1 vote
1 answer
118 views

Extracting word embedding features of a sentence using Transformer-XL

As you know, there are several pre-trained models that we can use to extract word embeddings. As an example, I can use the following codes to retrieve word2vec features of my text: ...
user avatar
  • 365
1 vote
0 answers
227 views

How to pass an array of variable length to the input of the neural network?

I have a bunch of two-dimensional points that I want to feed into my neural network as an input. Those points are positions of the visible obstacles around my agent. The main challenge is that the ...
user avatar
  • 111
1 vote
0 answers
44 views

Embeddings model: out of sample prediction with Keras for Collaborative Filtering

I have been trying to play with an example on Collaborative Filtering for Movie Recommendations (keras.io), which builds embedding layers for movies and users. Now, in a regular pre-trained word- and ...
user avatar
1 vote
1 answer
148 views

What's the best practice for dealing with OOV characters?

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
user avatar
1 vote
1 answer
46 views

What is the relationship between embeddings and deep metric learning?

Embeddings and deep metric learning seem architecturally identical. They both rely on using some hidden layer's vector representation of an input. What is the difference between the two? Are ...
user avatar
  • 941
4 votes
1 answer
920 views

What is embedding? (in the context of dimensionality reduction)

In the context of dimensionality reduction one often uses word embedding, which seems to me a rather technical mathematical term, which rather stands out compared to the rest of the discussion, which ...
user avatar
  • 1,578
1 vote
0 answers
1k views

Which dimensionality reduction technique works well for BERT sentence embeddings?

I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
user avatar
2 votes
0 answers
170 views

dimension of input layer for embeddings in Keras

It is not clear to me whether there is any difference between specifying the input dimension Input(shape=(20,)) or not ...
user avatar
  • 31
0 votes
0 answers
48 views

Is there an algorithm for placing 2-dimensional embeddings into a grid so they can be displayed?

I’m using PCA to reduce images down to 2d embeddings and I’d like to display the images in a grid. The Pudding did something like this with book covers, using tsne and a library called RasterFairy, by ...
user avatar
  • 113
1 vote
0 answers
60 views

Why are we interested in gradient with respect to input?

I am learning about sampling methods for Deep Embedding Learning. I was reading an article named: "Sampling Matters in Deep Embedding Learning" (https://arxiv.org/abs/1706.07567). In the ...
user avatar
  • 11
2 votes
1 answer
452 views

What is the difference between embedding in pure math and embedding in ML?

In ML the term "embedding" gets tossed around a lot and the term basically means the construction of a function that takes a high-dimensional vector to a low-dimensional vector in such a way ...
user avatar
1 vote
2 answers
33 views

Why are character level models considered less effective than word level models?

I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
user avatar
0 votes
1 answer
39 views

Why do Dense layers perform better than a mix of Conv Layers, Recurrent Layers on Sentiment Analysis with BERT emebddings?

I have used BERT to make embeddings out of the imdb review dataset and I am trying out some models to check their perfomance on sentiment analysis (0 for the bad reviews and 1 for the good ones). I ...
user avatar
  • 101
2 votes
1 answer
134 views

Is there any paper about applications of Deep Metric Learning on regression problem?

I'm trying to solve a problem in the field of transfer learning, more specifically, domain adaption where both the source domain and target domain are labeled. Basically it's to predict the ...
user avatar
  • 133
2 votes
1 answer
797 views

Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?

I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine): Note that (as is usually the case in Machine Learning) we ...
user avatar
1 vote
0 answers
39 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
user avatar
  • 11
6 votes
2 answers
1k views

Why BERT use learned positional embedding?

Compared with sinusoidal positional encoding used in Transformer, BERT's learned-lookup-table solution has 2 drawbacks in my mind: Fixed length Cannot reflect relative distance Could anyone please ...
user avatar
3 votes
1 answer
567 views

What are state of the art methods for creating embeddings for sets?

I want to create embeddings in $R^D$ for sets. So I want a function (probably a neural network) that takes in a set $ S = \{ s_1, \dots, s_n \} $ (and ideally of any size, so the number of elements ...
user avatar
1 vote
3 answers
58 views

General mathematical definition of a score

I understand what scores are in PCA, in particular this answer gives a good mathematical formulation: (Scores) are projections of the centred data in the linear space defined by the eigenvectors. ...
user avatar
0 votes
1 answer
156 views

Embedded markov chain example

I have an example of in my textbook of an "embeded markov chain", where I don't understand one step. Suppose that $(X_n)_{n\geq 0}$ is Markov$(\lambda, P)$. $\lambda$ is the initial distribution and ...
user avatar
1 vote
1 answer
95 views

Are the conditions of metric space satisfied in the latent space of a classification task?

Specifically, in the case of a neural network trained in a categorical classification task (cross-entropy loss function), does the final layer embedding space preserve the definition of distance ...
user avatar
  • 111
2 votes
0 answers
37 views

What are embeddings in the context of machine learning?

I would like to find out an intuitive explanation of what are embeddings in the context of machine learning and neural networks. Is that essentially the same thing as a manifold? I've read a bunch of ...
user avatar
0 votes
0 answers
184 views

Using label encoder on a categorical feature that we want to embed

I have a dataset with feature that have very high cardinality, doing one-hot encoding is not an option because of memory limitations, so I am currently label encoding this feature and then I feed that ...
user avatar
1 vote
0 answers
49 views

t-sne embedding to medium-dimensions (e.g. 100 dimensions)?

I am using t-sne on 252 dimensional data to embed to lower-dimensions. I am curious to know if it is academically justifiable to embed it into medium dimensions such as 100 dimensions, or 80 ...
user avatar