Questions tagged [embeddings]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
5
votes
1answer
476 views

What is the intuition behind the positional cosine encoding in the transformer network?

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence. ...
5
votes
2answers
954 views

How to embed in Euclidean space

I have what I think might be a standard machine learning problem but I can't find a clear solution. I have lots vectors of different dimensions. For each pair of vectors I can compute their ...
5
votes
1answer
292 views

What is the difference in the latent space of a variational autoencoder and a regular autoencoder?

Should VAEs be even used for non-generative tasks? If I were to use both models for embedding images, how would the embedding space differ on a structural level?
4
votes
0answers
303 views

Why researchers use conv1d for embeddings instead of dense layers?

In some papers (like Reinforcement learning for Vehicle Routing Problem), researchers use conv1d to embed the problem input into a hyperspace; for example, in solving TSP, they use conv1d on the (x,y) ...
3
votes
1answer
450 views

Facebook's infersent intuition

When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with: (u, v) the ...
3
votes
2answers
3k views

How to use “IDs” as an input variable to a ML model?

I am trying to include a variable like "account number" which is an "ID" as a predictive variable for a logistic regression model. In fact there are several columns in my dataset that are "IDs" but ...
2
votes
1answer
42 views

What is the difference between embedding in pure math and embedding in ML?

In ML the term "embedding" gets tossed around a lot and the term basically means the construction of a function that takes a high-dimensional vector to a low-dimensional vector in such a way ...
2
votes
1answer
856 views

Is the Keras Embedding layer dependent on the target label?

I learned how to 'use' the Keras Embedding layer, but I am not able to find any more specific information about the actual behavior and training process of this layer. For now, I understand that the ...
2
votes
2answers
286 views

How to rank products using deep learning for recommender systems?

I am going to implement a recommender system based on this paper. It basically uses a double embedding technique, one for the user representation and another one for the products (movies, clothes, ...
2
votes
1answer
27 views

Is there any theory on the order of Autoregression model for periodic time series? [closed]

Say M periodic signals, then one can safely say using AR-M model can achieve the perfect prediction. But how about further, in a more general sense, is there any publications on this? Update: Here ...
2
votes
0answers
62 views

Why BERT use learned positional embedding?

Compared with sinusoidal positional encoding used in Transformer, BERT's learned-lookup-table solution has 2 drawbacks in my mind: Fixed length Cannot reflect relative distance Could anyone please ...
2
votes
0answers
25 views

What are embeddings in the context of machine learning?

I would like to find out an intuitive explanation of what are embeddings in the context of machine learning and neural networks. Is that essentially the same thing as a manifold? I've read a bunch of ...
2
votes
0answers
12 views

Neural network embedding layers allowing multiple class-membership features

Is there a version of embedding layers for neural networks that allows for multiple class-membership features? Any frameworks that have implemented this? E.g. imagine we are trying to predict ...
2
votes
0answers
21 views

How to apply the diffusion maps when the matrix is PSD but not positivity preserving?

In order to apply the diffusion maps in a matrix $C\in\mathbb R^{n\times n}$ , that matrix must obey some restrictions, C is symmetric: $C_{ij} = C_{ji}$, C is positivity preserving (PP): $\forall ...
2
votes
0answers
88 views

Can you use VAEs to produce deep word embeddings?

There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
2
votes
0answers
88 views

Recovering a distance matrix from nonnegative sparse correlation matrix?

After doing extensive literature research in all sorts of science I am completely puzzled. I am trying to find out what the state-of-the-art techniques would be to recover a (let's say euclidean) ...
2
votes
0answers
148 views

Is there an extension of PCA for data embedded in hyperbolic spaces?

I'm working on a project where we are embedding data into an n-dimensional Poincare ball similar to this paper. However, we'd like to take the additional step of reducing this data to a 2-dimensional ...
1
vote
3answers
49 views

General mathematical definition of a score

I understand what scores are in PCA, in particular this answer gives a good mathematical formulation: (Scores) are projections of the centred data in the linear space defined by the eigenvectors. ...
1
vote
1answer
81 views

What are state of the art methods for creating embeddings for sets?

I want to create embeddings in $R^D$ for sets. So I want a function (probably a neural network) that takes in a set $ S = \{ s_1, \dots, s_n \} $ (and ideally of any size, so the number of elements ...
1
vote
2answers
29 views

Why are character level models considered less effective than word level models?

I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
1
vote
1answer
465 views

How do I add a missing word to a pretrained embedding?

I have a pretrained word embedding and want to add missing words to it. How exactly should I do that? I think to just randomly initialize the vector is not a good idea. I heard something about ...
1
vote
0answers
16 views

Is there any paper about applications of Deep Metric Learning on regression problem?

I'm trying to solve a problem in the field of transfer learning, more specifically, domain adaption where both the source domain and target domain are labeled. Basically it's to predict the ...
1
vote
0answers
16 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
1
vote
1answer
35 views

Are the conditions of metric space satisfied in the latent space of a classification task?

Specifically, in the case of a neural network trained in a categorical classification task (cross-entropy loss function), does the final layer embedding space preserve the definition of distance ...
1
vote
0answers
42 views

t-sne embedding to medium-dimensions (e.g. 100 dimensions)?

I am using t-sne on 252 dimensional data to embed to lower-dimensions. I am curious to know if it is academically justifiable to embed it into medium dimensions such as 100 dimensions, or 80 ...
1
vote
0answers
43 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
0answers
32 views

how to use contextual embeddings?

I've read about pretrained word embeddings, and I understand how to use them. Basically, if I have the word nail (for example), there is a look up table where I can use the embedding for that word. ...
1
vote
0answers
40 views

Are training-loss optimised embeddings useless? (help resolve a disagreement)

The aim We are training a feed forward neural network as a regressor, with the aim of using the activations of the final layer as a type of embedding vector to represent the input examples. The ...
1
vote
0answers
207 views

Machine learning on JSON/XML/DOM data

What's the best approach for machine learning on deeply hierarchical JSON/XML/DOM documents (not counting text nodes)? Say I want to recognize and generate documents similar to a training set of ...
1
vote
0answers
324 views

feature embedding for categorical features

I'm training a model and among the features, I have the language of the users. Right now I have done one-hot encoding on the language feature. But I think it would make more sense to have the language ...
1
vote
0answers
88 views

Derivation of local linear embedding

Might be a trivial question, but how do I solve for the following constrained optimization problem that appeared in local linear embedding? $$\min_{w_1,\cdots,w_k} \|x-\sum_{i=1}^k w_i x_i\|^2 \text{ ...
0
votes
1answer
52 views

Embedded markov chain example

I have an example of in my textbook of an "embeded markov chain", where I don't understand one step. Suppose that $(X_n)_{n\geq 0}$ is Markov$(\lambda, P)$. $\lambda$ is the initial distribution and ...
0
votes
1answer
121 views

Is it possible to use seq2Seq models to predict HTML code from XML file?

I have XML file that describes some embedded components. So the file has different markups that correspond to different fields. The intention behind this project is to generate automatically UI ...
0
votes
1answer
29 views

Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?

I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine): Note that (as is usually the case in Machine Learning) we ...
0
votes
0answers
12 views

Clustering & classification of customers

I have three datasets : one about general population, one about customers for a specific brand and then one with people that were part of an advertisement event and whether or that person converted to ...
0
votes
1answer
14 views

Why do Dense layers perform better than a mix of Conv Layers, Recurrent Layers on Sentiment Analysis with BERT emebddings?

I have used BERT to make embeddings out of the imdb review dataset and I am trying out some models to check their perfomance on sentiment analysis (0 for the bad reviews and 1 for the good ones). I ...
0
votes
0answers
4 views

What is gravity in the context of “folding” in recommender systems?

What is "gravity" in the context of recommender systems? More specifically, how is it supposed to help with the "folding" problem where irrelevant queries may be returned if we don't provide ...
0
votes
0answers
19 views

Calculating similarities between two populations using embeddings

I would like to find items from population B that are most similar to an item from population A. I have the following set up: Two sparse datasets where each row is an item (treat row index as item ID)...
0
votes
0answers
8 views

Integrate popularity with the approximate nearest neighbor searching?

I studied the mechanism of some ANN algorithms but only find that each stored vector is treated equally. That is, the popularity of the corresponding vectors are ignored. How can all vector ...
0
votes
0answers
26 views

What's the best way to combine embeddings for ID list features?

I am using an embedding table to incorporate a high cardinality categorical feature into a model. The tricky part is that for 1 training observation this feature may have multiple values. For example, ...
0
votes
0answers
158 views

Embedding layer before LSTM layer

I am re-creating a clustering and churn prediction framework, cluschurn, which they deployed in production at Snap, Inc. In their research paper, paper_link, they use 14 days of user data and treat it ...
0
votes
0answers
27 views

Categorical Variables Encoding of Sets

I'm facing a problem in which I need to encode a categorical variable, which can take several values at the same time (and is basically a set), as input for a classifier. For instance, assuming the ...
0
votes
0answers
12 views

Is optimizing an embedding a convex or non-convex process?

Suppose we have input data with several thousand one-hot dimensions per element, representing, say, words in a passage of text. An embedding layer is a common feature at the top of machine learning ...
0
votes
0answers
6 views

How to create linear user embedding from some answers to binary questions?

I have each user U_i answering 10 binary questions out of a pool Q with either answer 1 or 2. I would like to learn an embedding of user profiles based on these answers to predict is answer to other ...
0
votes
0answers
52 views

Using label encoder on a categorical feature that we want to embed

I have a dataset with feature that have very high cardinality, doing one-hot encoding is not an option because of memory limitations, so I am currently label encoding this feature and then I feed that ...
0
votes
0answers
11 views

Signal Embeddings using the skip-gram or CBOW model

So my work involves looking at a bunch of waveforms in the context of classifying events. I often am looking for new ways to represent my waveforms, and in my searching, I came across audio embeddings ...
0
votes
1answer
14 views

Reconstructing face from randomised embedding

It is fairly agreed in literature that from a given face-embedding (that is a vector of features values) it is possible, with a good amount of effort, to reconstruct the original face, (See here for ...
0
votes
1answer
82 views

Spectral embedding: interpretation of new dimensions

I'm trying to gain an intuition for the 2nd dimension in the spectral embedding of an S-shaped dataset as in this example: The 1st dimension seems to neatly capture the local similarity between ...
0
votes
2answers
2k views

Learning image embeddings using VGG and Word2Vec

Background: In word2vec we pass in a one-hot encoding of our target word into a simple neural network which is trained to predict context words from a window around our target. We eventually take the ...
0
votes
1answer
33 views

How to calculate accuracy of identification using embedings and corresponded labels? [closed]

I'm trying to implement deep speaker embeding system and after getting voice embeddings I need to somehow calculate accuracy. But there is no mention in deep speaker paper about how they calculated ...