Questions tagged [embeddings]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
1answer
126 views

Why BERT use learned positional embedding?

Compared with sinusoidal positional encoding used in Transformer, BERT's learned-lookup-table solution has 2 drawbacks in my mind: Fixed length Cannot reflect relative distance Could anyone please ...
0
votes
2answers
2k views

Learning image embeddings using VGG and Word2Vec

Background: In word2vec we pass in a one-hot encoding of our target word into a simple neural network which is trained to predict context words from a window around our target. We eventually take the ...
3
votes
1answer
30 views

What is embedding? (in the context of dimensionality reduction)

In the context of dimensionality reduction one often uses word embedding, which seems to me a rather technical mathematical term, which rather stands out compared to the rest of the discussion, which ...
0
votes
0answers
24 views

Which dimensionality reduction technique works well for BERT sentence embeddings?

I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
0
votes
0answers
12 views

dimension of input layer for embeddings in Keras

It is not clear to me whether there is any difference between specifying the input dimension Input(shape=(20,)) or not ...
0
votes
0answers
21 views

Is there an algorithm for placing 2-dimensional embeddings into a grid so they can be displayed?

I’m using PCA to reduce images down to 2d embeddings and I’d like to display the images in a grid. The Pudding did something like this with book covers, using tsne and a library called RasterFairy, by ...
1
vote
0answers
27 views

Why are we interested in gradient with respect to input?

I am learning about sampling methods for Deep Embedding Learning. I was reading an article named: "Sampling Matters in Deep Embedding Learning" (https://arxiv.org/abs/1706.07567). In the ...
2
votes
1answer
53 views

What is the difference between embedding in pure math and embedding in ML?

In ML the term "embedding" gets tossed around a lot and the term basically means the construction of a function that takes a high-dimensional vector to a low-dimensional vector in such a way ...
0
votes
0answers
13 views

Clustering & classification of customers

I have three datasets : one about general population, one about customers for a specific brand and then one with people that were part of an advertisement event and whether or that person converted to ...
1
vote
2answers
31 views

Why are character level models considered less effective than word level models?

I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
0
votes
1answer
16 views

Why do Dense layers perform better than a mix of Conv Layers, Recurrent Layers on Sentiment Analysis with BERT emebddings?

I have used BERT to make embeddings out of the imdb review dataset and I am trying out some models to check their perfomance on sentiment analysis (0 for the bad reviews and 1 for the good ones). I ...
1
vote
0answers
19 views

Is there any paper about applications of Deep Metric Learning on regression problem?

I'm trying to solve a problem in the field of transfer learning, more specifically, domain adaption where both the source domain and target domain are labeled. Basically it's to predict the ...
0
votes
0answers
4 views

What is gravity in the context of “folding” in recommender systems?

What is "gravity" in the context of recommender systems? More specifically, how is it supposed to help with the "folding" problem where irrelevant queries may be returned if we don't provide ...
5
votes
1answer
334 views

What is the difference in the latent space of a variational autoencoder and a regular autoencoder?

Should VAEs be even used for non-generative tasks? If I were to use both models for embedding images, how would the embedding space differ on a structural level?
0
votes
1answer
45 views

Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?

I recently re-read Stanford CS231N lecture notes on computer vision and backpropagation, and I came across this passage (emphasis mine): Note that (as is usually the case in Machine Learning) we ...
1
vote
0answers
19 views

Adam converges while SGD does not improve at all

I am trying to build a model based movie recommendation system with a neural network. The architecture looks as follows: ...
0
votes
0answers
19 views

Calculating similarities between two populations using embeddings

I would like to find items from population B that are most similar to an item from population A. I have the following set up: Two sparse datasets where each row is an item (treat row index as item ID)...
0
votes
0answers
8 views

Integrate popularity with the approximate nearest neighbor searching?

I studied the mechanism of some ANN algorithms but only find that each stored vector is treated equally. That is, the popularity of the corresponding vectors are ignored. How can all vector ...
1
vote
1answer
110 views

What are state of the art methods for creating embeddings for sets?

I want to create embeddings in $R^D$ for sets. So I want a function (probably a neural network) that takes in a set $ S = \{ s_1, \dots, s_n \} $ (and ideally of any size, so the number of elements ...
0
votes
0answers
28 views

What's the best way to combine embeddings for ID list features?

I am using an embedding table to incorporate a high cardinality categorical feature into a model. The tricky part is that for 1 training observation this feature may have multiple values. For example, ...
0
votes
0answers
250 views

Embedding layer before LSTM layer

I am re-creating a clustering and churn prediction framework, cluschurn, which they deployed in production at Snap, Inc. In their research paper, paper_link, they use 14 days of user data and treat it ...
0
votes
0answers
13 views

Is optimizing an embedding a convex or non-convex process?

Suppose we have input data with several thousand one-hot dimensions per element, representing, say, words in a passage of text. An embedding layer is a common feature at the top of machine learning ...
1
vote
1answer
502 views

How do I add a missing word to a pretrained embedding?

I have a pretrained word embedding and want to add missing words to it. How exactly should I do that? I think to just randomly initialize the vector is not a good idea. I heard something about ...
1
vote
3answers
51 views

General mathematical definition of a score

I understand what scores are in PCA, in particular this answer gives a good mathematical formulation: (Scores) are projections of the centred data in the linear space defined by the eigenvectors. ...
0
votes
1answer
53 views

Embedded markov chain example

I have an example of in my textbook of an "embeded markov chain", where I don't understand one step. Suppose that $(X_n)_{n\geq 0}$ is Markov$(\lambda, P)$. $\lambda$ is the initial distribution and ...
1
vote
1answer
41 views

Are the conditions of metric space satisfied in the latent space of a classification task?

Specifically, in the case of a neural network trained in a categorical classification task (cross-entropy loss function), does the final layer embedding space preserve the definition of distance ...
2
votes
0answers
26 views

What are embeddings in the context of machine learning?

I would like to find out an intuitive explanation of what are embeddings in the context of machine learning and neural networks. Is that essentially the same thing as a manifold? I've read a bunch of ...
0
votes
0answers
6 views

How to create linear user embedding from some answers to binary questions?

I have each user U_i answering 10 binary questions out of a pool Q with either answer 1 or 2. I would like to learn an embedding of user profiles based on these answers to predict is answer to other ...
0
votes
0answers
74 views

Using label encoder on a categorical feature that we want to embed

I have a dataset with feature that have very high cardinality, doing one-hot encoding is not an option because of memory limitations, so I am currently label encoding this feature and then I feed that ...
2
votes
2answers
288 views

How to rank products using deep learning for recommender systems?

I am going to implement a recommender system based on this paper. It basically uses a double embedding technique, one for the user representation and another one for the products (movies, clothes, ...
1
vote
0answers
42 views

t-sne embedding to medium-dimensions (e.g. 100 dimensions)?

I am using t-sne on 252 dimensional data to embed to lower-dimensions. I am curious to know if it is academically justifiable to embed it into medium dimensions such as 100 dimensions, or 80 ...
5
votes
2answers
1k views

How to embed in Euclidean space

I have what I think might be a standard machine learning problem but I can't find a clear solution. I have lots vectors of different dimensions. For each pair of vectors I can compute their ...
0
votes
0answers
16 views

Signal Embeddings using the skip-gram or CBOW model

So my work involves looking at a bunch of waveforms in the context of classifying events. I often am looking for new ways to represent my waveforms, and in my searching, I came across audio embeddings ...
2
votes
0answers
12 views

Neural network embedding layers allowing multiple class-membership features

Is there a version of embedding layers for neural networks that allows for multiple class-membership features? Any frameworks that have implemented this? E.g. imagine we are trying to predict ...
2
votes
0answers
55 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
0answers
35 views

how to use contextual embeddings?

I've read about pretrained word embeddings, and I understand how to use them. Basically, if I have the word nail (for example), there is a look up table where I can use the embedding for that word. ...
0
votes
1answer
22 views

Reconstructing face from randomised embedding

It is fairly agreed in literature that from a given face-embedding (that is a vector of features values) it is possible, with a good amount of effort, to reconstruct the original face, (See here for ...
1
vote
0answers
331 views

feature embedding for categorical features

I'm training a model and among the features, I have the language of the users. Right now I have done one-hot encoding on the language feature. But I think it would make more sense to have the language ...
2
votes
1answer
944 views

Is the Keras Embedding layer dependent on the target label?

I learned how to 'use' the Keras Embedding layer, but I am not able to find any more specific information about the actual behavior and training process of this layer. For now, I understand that the ...
2
votes
0answers
22 views

How to apply the diffusion maps when the matrix is PSD but not positivity preserving?

In order to apply the diffusion maps in a matrix $C\in\mathbb R^{n\times n}$ , that matrix must obey some restrictions, C is symmetric: $C_{ij} = C_{ji}$, C is positivity preserving (PP): $\forall ...
0
votes
1answer
86 views

Spectral embedding: interpretation of new dimensions

I'm trying to gain an intuition for the 2nd dimension in the spectral embedding of an S-shaped dataset as in this example: The 1st dimension seems to neatly capture the local similarity between ...
6
votes
1answer
576 views

What is the intuition behind the positional cosine encoding in the transformer network?

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence. ...
2
votes
0answers
90 views

Can you use VAEs to produce deep word embeddings?

There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
1
vote
0answers
40 views

Are training-loss optimised embeddings useless? (help resolve a disagreement)

The aim We are training a feed forward neural network as a regressor, with the aim of using the activations of the final layer as a type of embedding vector to represent the input examples. The ...
2
votes
1answer
27 views

Is there any theory on the order of Autoregression model for periodic time series? [closed]

Say M periodic signals, then one can safely say using AR-M model can achieve the perfect prediction. But how about further, in a more general sense, is there any publications on this? Update: Here ...
3
votes
1answer
487 views

Facebook's infersent intuition

When reviewing Infersent's architecture here, I noticed that, after encoding the premise and hypothesis to obtain two vectors u and v, they feed the set of fully connected layers with: (u, v) the ...
1
vote
0answers
89 views

Derivation of local linear embedding

Might be a trivial question, but how do I solve for the following constrained optimization problem that appeared in local linear embedding? $$\min_{w_1,\cdots,w_k} \|x-\sum_{i=1}^k w_i x_i\|^2 \text{ ...
4
votes
0answers
309 views

Why researchers use conv1d for embeddings instead of dense layers?

In some papers (like Reinforcement learning for Vehicle Routing Problem), researchers use conv1d to embed the problem input into a hyperspace; for example, in solving TSP, they use conv1d on the (x,y) ...
2
votes
0answers
89 views

Recovering a distance matrix from nonnegative sparse correlation matrix?

After doing extensive literature research in all sorts of science I am completely puzzled. I am trying to find out what the state-of-the-art techniques would be to recover a (let's say euclidean) ...
0
votes
1answer
126 views

Is it possible to use seq2Seq models to predict HTML code from XML file?

I have XML file that describes some embedded components. So the file has different markups that correspond to different fields. The intention behind this project is to generate automatically UI ...