10
votes
Accepted
What is the difference between embedding in pure math and embedding in ML?
In pure mathematics an embedding is any function $f\colon X \to Y$ that is injective and structure-preserving. What do these terms mean?
Injective
Different elements of $X$ are always mapped to ...
- 620
10
votes
What is the intuition behind the positional cosine encoding in the transformer network?
In positional encoding you encode the dimension with different frequency waves. Together with a position (on this wave) this gives you encoding that corresponds to each input. The encoding is ...
- 62.3k
9
votes
Accepted
Embedding data into a larger dimension space
As Forrest mentioned embedding data into a higher dimension (sometimes called basis expansion) is a common method which allows a linear classifier to observe a non-linear input space. Examples are ...
- 1,178
7
votes
Accepted
Facebook's infersent intuition
First of all, many tricks in deep learning are used because they were "proved to work", with post factum theoretical rationalizations. So in many cases the "why" questions can be only answered in ...
- 141k
6
votes
How to use "IDs" as an input variable to a ML model?
ID variables like phone number should not be included as predictors, because you are trying to train a model to understand general patterns. Phone number doesn't offer the model any real insight into ...
- 1,782
5
votes
If the curse of dimensionality exists, how does embedding search work?
The origion of vector space model is as follows:
The idea that the meaning of a word might be modeled as a point in a multi-
dimensional semantic space came from psychologists like Charles E. Osgood, ...
- 6,912
4
votes
What are state of the art methods for creating embeddings for sets?
Indeed, in the last to-three years there have been some important publications on this topic. I do not know all of them, and cannot give a complete survey of the current status. One important paper ...
- 14.3k
4
votes
How to embed in Euclidean space
These are known as multidimensional scaling algorithms. From wikipedia (https://en.wikipedia.org/wiki/Multidimensional_scaling), "An MDS algorithm aims to place each object in N-dimensional space ...
- 369
3
votes
What is embedding? (in the context of dimensionality reduction)
The word is used ambiguously, e.g. this quote from Google crash course on machine learning says [my comments in square brackets]:
An embedding is a relatively low-dimensional space [subspace] into ...
- 141k
3
votes
Accepted
Is the Keras Embedding layer dependent on the target label?
Embeddings layer for vocabulary of size $m$, that encodes each word into embeddings vector of size $k$ is a shorthand for having the words one-hot encoded using into $m$ features and then putting ...
- 141k
3
votes
Generating embeddings for languages without a written representation?
Embeddings, as other neutral networks, need huge datasets. People usually scrape thousands of books, news articles and web pages for that. For audio files, this would mean thousands of hours of ...
- 141k
3
votes
ArcFace - How to compute $\cos(t+m)$ when $t+m > \pi$
Using the sum-to-product formulae for trigonometric functions you have the exact equation:
$$\begin{align}
\cos (t+m)
&= \cos ((t+\tfrac{m}{2})+\tfrac{m}{2}) \\[6pt]
&= \cos((t+\tfrac{m}{2})-\...
- 133k
3
votes
Accepted
Extracting word embedding features of a sentence using Transformer-XL
I'm not sure if you're asking specifically about gensim, and this question will likely be closed (generally questions about specific libraries aren't accepted) but ...
- 526
3
votes
Is it possible to use seq2Seq models to predict HTML code from XML file?
I haven't seen any examples to predict code using encoder-decoder methods (but you probably can!). However, "The Unreasonable Effectiveness of Recurrent Neural Networks" by Andrej Karpathy outlines ...
- 94k
2
votes
What is the difference in the latent space of a variational autoencoder and a regular autoencoder?
For the vanilla autoencoder the structure is like this:
It can be treated as a nonlinear extension of PCA, while for the variational autoencoder a mean and a standard deviation is added as a layer ...
- 6,912
2
votes
How to use "IDs" as an input variable to a ML model?
With new functionalities implemented in tensorflow such as hashing and embedding, I was able to take advantage of ID variables in my data and use them as predictive variables. You can find the ...
- 201
2
votes
How to embed in Euclidean space
vectors that are similar under the original measure have small
Euclidean distance under the embedding
This is the goal of dimensionality reduction, especially the nonlinear dimensionality reduction,...
- 5,986
2
votes
How to rank products using deep learning for recommender systems?
A common approach in practice is to first filter the dataset with a light model to generate a candidate set, and then apply the heavy model only to those candidates. This approach is used, for example,...
- 6,242
2
votes
Is there any theory on the order of Autoregression model for periodic time series?
If you have a perfectly predictable signal periodic in frequency $f$, the pattern repeats every cycle, or $1/f$ time units.
You need a single lag to predict it, the series lagged by $1/f$ time units. ...
- 19.5k
2
votes
Accepted
General mathematical definition of a score
I don't know about a score definition that "rules them all," but consider the notion shown in here https://www.ncbi.nlm.nih.gov/pubmed/28715259. There, the first PC score is defined as a linear ...
- 6,001
2
votes
Accepted
What's the best practice for dealing with OOV characters?
Having a special unknown token is a common practice regardless of whether you work with words or characters.
However, note that the model needs to learn how to handle the unknown characters. If any ...
- 3,459
2
votes
Accepted
Is there any paper about applications of Deep Metric Learning on regression problem?
You could have a look at the MLKR algorithm (Metric Learning for Kernel Regression): although the basic version learns a Mahalanobis metric (linear transformation), it should be easily adaptable to be ...
2
votes
Neural network backpropagation to update inputs, not weights (e.g. fine-tuning embeddings)?
If you view the back-propagation purely as a graph algorithm operating on the computation graph, the inputs are qualitatively the same nodes as the parameters - some floats that one can change and the ...
- 3,459
2
votes
Why BERT use learned positional embedding?
Fixed length
BERT, same as Transformer, use attention as a key feature. The attention as used in those models, has a fixed span as well.
Cannot reflect relative distance
We assume neural networks ...
- 141k
2
votes
Accepted
Embedded markov chain example
The key realization is that because the transition matrix is constant, if you start at state $i$, the probability that at the next stage you are in any given state $j$ is independent of the time ...
- 41k
2
votes
How do I go from embeddings to queries, keys and values in the Transformer model?
The queries, keys and values are computed from the embeddings $x$ using matrix multiplication:
\begin{align}
q &= xW_Q \\
k &= xW_K \\
v &= xW_V,
\end{align}
where $W_Q$, $W_K$ and $W_V$ ...
- 141
1
vote
Why are character level models considered less effective than word level models?
Character sequences are much longer than word sequences. This is even more critical with Transformer models that require quadratic memory with respect to the sequence length.
Also, word embeddings are ...
- 3,459
1
vote
t-sne embedding to medium-dimensions (e.g. 100 dimensions)?
WHY?
A troubling point with t-SNE is that it does not give a functional relationship between the high-dimension space and the reduced-dimension space, limiting the usefulness of t-SNE. For instance, ...
- 67.1k
1
vote
t-sne embedding to medium-dimensions (e.g. 100 dimensions)?
Here's an idea I am currently pursuing. I have large biological data sets (in about 20,000 dimensions, one for each gene). The mental model is that the actual data lives on (or at least near) a ...
1
vote
Why BERT use learned positional embedding?
Here is my current understanding to my own question.
It probably related BERT's transfer learning background. The learned-lookup-table indeed increase learning effort in pretrain stage, but the extra ...
- 547
Only top scored, non community-wiki answers of a minimum length are eligible