Questions tagged [manifold-learning]
Manifold learning subsumes techniques conceived for problems where data of interest are assumed to lie on an embedded non-linear manifold within a higher-dimensional space.
89
questions
0
votes
0
answers
16
views
Why do we expect the eigenvalues of the Gramian matrix found by Maximum Variance Unfolding to capture the number of degrees of freedom in the data?
Maximum Variance Unfolding (MVU) is a manifold learning method which, like other forms of dimensionality reduction, makes the assumption that whatever (high-dimensional) data we're dealing with "...
- 169
0
votes
0
answers
36
views
Is it correct to do SVD from the latent space of an autoencoder?
Is it correct to do SVD from the latent space of an autoencoder?
I am asking because I think that by performing SVD from a latent space, and plotting the singular values, it is possible to know the ...
- 1
0
votes
0
answers
26
views
Applying non-linear dimensionality reduction on binary data
Would be beneficial to apply nonlinear dimensionality reduction on a binary dataset (around 200 binary features) ? what is difference from applying MCA (multiple correspondence analysis).
- 11
4
votes
0
answers
195
views
Explanation of UMAP assumptions
Uniform Manifold Approximation (UMAP) is a technique for dimensionality reduction and visualization. The author of UMAP states that the algorithm is founded on three assumptions about the data:
The ...
- 4,080
3
votes
2
answers
161
views
What is the relationship between noise reduction and dimension reduction?
My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data ...
- 53
2
votes
1
answer
72
views
Determine if high dimensional data is multimodal
I have p-dimensional data and I need to determine if that data has significant modes or if it’s clustered in any way. Here p=50, (dense embedding), we have n samples and p <<< n.
What are ...
- 21
2
votes
1
answer
82
views
Why does data get so tangled up in high dimension?
When I look at textbooks on classification and machine learning, many of the examples focus on data that is often twisted up such as to avoid linear separation. I have an example picture below. The ...
- 1,502
0
votes
0
answers
137
views
Reproducing kernel hilbert space norm as smoothness functional
Let $K:X \times X \rightarrow \mathbb{R}$ be a Mercer kernel with an associated RKHS $H$ then the norm $|f|_H^2$ can be used as a way to ensure that $f$ is smooth in $H$. If i understand correctly, ...
- 153
2
votes
0
answers
33
views
Any theory on whether good choices of $k$ depend on $N$ and $D$ in KNN classification?
I am well aware that cross validation is a usual method for selecting hyperparameters. However, I am looking for theoretical guidance on how to pick $k$, the number of neighbors, for a $k$-nearest-...
- 641
0
votes
1
answer
496
views
Variance used in t-SNE
I was going through the t-SNE, but I am bit confused. While the original paper of t-SNE is based on the SNE and SNE uses $\sigma_i^2$ (note the subscript $i$) while calculating the similarity of point ...
1
vote
0
answers
29
views
Analytical tools to analyze the characteristics of a data manifold
In the paper "Emergence of separable manifolds in deep language representations," the authors use an analytical tool called Mean Field Theoretic Manifold Analysis to measure the manifold ...
- 11
11
votes
4
answers
1k
views
Dimension reduction using space filling curve to avoid "Curse of dimensionality"?
In machine learning, we want to train a model. While training, if the dimension of data is high, we have a problem (Curse of Dimensionality), so we want to reduce the dimension of our data.
Since we ...
- 111
1
vote
2
answers
636
views
What validation if KFold scores differ a lot? Repeated KFold, LOO or Holdout?
Suppose you are given a medium-sized dataset and you did a KFold validation once. You notice that scores on each old differ noticeably. Which validation type is the most practical?
I thought about ...
2
votes
0
answers
42
views
State-of-the-art methods for out-of-sample-extension
I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis.
I've been using the Nystrom method for this task, and some ...
- 799
0
votes
1
answer
31
views
Simulate data that rotate on itself like DNA
I'm doing an exercise on the reduction of nonlinear dimension in manifold. I want to use LLE for that.
And I'd like to simulate data in the form of DNA, so it rotates on itself. That is, a flat ...
- 121
1
vote
0
answers
170
views
how do i calculate (and Apply) Loss gradients with respect to the input (not the weights) of a CNN?
I have a trained generator, i would like to apply a loss function to the output and optimize the input (latent vector) using a gradient decent optimizer.
i don't know how to calculate the gradients ...
- 11
1
vote
0
answers
149
views
Manifold Hypothesis vs. Latent Variables Assumption [closed]
As I understand:
The manifold hypothesis claims that real world data, although represented in high dimension space, actually lies on a manifold in that space. I.e. that the actual data structure is of ...
- 3,181
0
votes
1
answer
908
views
Multivariate Jensen-Shannon divergence
This paper says multivariate Jensen-Shannon divergence is
$$JS(\mathbf{p}_1,\dots,\mathbf{p}_K) = \frac{1}{m} \sum KL(\mathbf{p}_i || \bar{\mathbf{p}})$$
with $KL$ being the KL-divergence of the ...
- 3,481
1
vote
0
answers
602
views
Differentiable PCA? [closed]
Is there a differentiable method for dimensionality reduction that is either based on PCA or has the properties of:
Mathematically or algorithmically defined, e.g. not trained like an ML model or t-...
- 251
2
votes
0
answers
61
views
PCA for non linear manifolds - Yann Lecun, Deep Learning Course, Question from a Lecture
I was watching the following lecture and at the very end of it, one of the students asked LeCun about using PCA for expression and pose feature extraction.
https://www.youtube.com/watch?v=0bMe_vCZo30&...
1
vote
1
answer
394
views
Testing of hypothesis for the linearity of a data? PCA suggested, but how do we design a statistical test using it? [closed]
Suppose we're given the data set $\{x_1 \dots x_n\}$ in $\mathbb{R}^D$ the $D$-dimensional Euclidean space, and assume this data has intrinsic dimension $d < D.$ N.B. this just means that data is ...
- 721
3
votes
1
answer
1k
views
Meaning of "Manifold of interest"
While reading a paper, I stumble upon the following sentence and couldn't figure out its meaning:
Informally, for an input set of real images, we say that the set of layer activations (for any ...
- 137
2
votes
0
answers
37
views
Theoretical justification behind assuming that the data is locally uniformly distributed, as seem to be used by manifold learning community
In at least three or more papers I've been studying that introduced novel algorithms for the estimation of intrinsic dimensionality (ID) based on nearest neighborhood (NN) techniques, I observed that ...
- 721
1
vote
0
answers
20
views
Is there any manifold learning literature where the both the ambient dimension and intrinsic dimension of the data are high?
I'm new to manifold learning, and from my understanding it normally denotes a subject which starts with the assumption that the data $\{x_1 \dots x_n\} \subset (M,g) \subset \mathbb{R}^p,$ where $(M,g)...
- 721
0
votes
0
answers
53
views
When is the manifold hypothesis violated?
If the high-dimensional dataset is of full rank, like an N samples by N features matrix, will the manifold hypothesis be violated? i.e. we cannot assume those data live on a lower-dimensional space?
- 61
2
votes
0
answers
306
views
Does classic MultiDimensional Scaling reconstructs data upto a rotation when there's no noise and embedding dimensions equals original data dimension?
In the setup of classical MultiDimensional Scaling (MDS), assume that $D:=[d_{ij}]$ be an $n \times n$ distance matrix, i.e. $d(i,i)=0, d(i,j)=d(j,i) > 0 \forall i, j = 1 \dots n.$ Assume that: ...
- 721
4
votes
0
answers
118
views
In Multidimensional Scaling (MDS), is it safe to assume that the optimal embedding dimension grows with the growth of sample size?
My question is more of a theoretical nature, so it'd be great to have some references to papers, but it'd be also nice to see some experiments.
Let $D:=[d_{ij}]$ be an $n \times n$ distance matrix, i....
- 721
1
vote
0
answers
93
views
How does high dimensional data residing in low dimensional manifold have high variance?
I was pondering over the fact that for a very high dimensional data, if it lies in a low dimesional manifold, then in the bias variance decomposition for it, the variance will be high and bias will be ...
- 21
1
vote
0
answers
32
views
Why RANDOM noise images always predicted as BIRD?
Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%.
However when feeding into 5000 random noise images (Gaussian noise with the ...
- 11
0
votes
0
answers
48
views
Manifold optimization theory
Does there exist some elementary book for optimization on manifolds for problems in finance/economics?
0
votes
1
answer
185
views
Spectral embedding: interpretation of new dimensions
I'm trying to gain an intuition for the 2nd dimension in the spectral embedding of an S-shaped dataset as in this example:
The 1st dimension seems to neatly capture the local similarity between ...
- 101
0
votes
1
answer
48
views
Linear and Non-Linear dimensionality reduction with missing variables
I'm trying to compare the two types of reduction by applying it to a list of ingredients for about 1000 sponge cakes.
The ingredient lists do however have miss certain ingredients out for some cakes, ...
- 23
2
votes
1
answer
606
views
Dimensionality Reduction for Optimally Preserving KNN
Do any dimensionality reduction techniques find embeddings which optimally preserve the K-nearest neighbors of each point? If no algorithm provably does this, are there algorithms which heuristically ...
- 21
4
votes
1
answer
124
views
Quantifying a manifold folding unto itself
I have a dataset of ~7k scattered points in 3D which represents a manifold that may or may not "fold unto itself". Here's an example where this does happen (look at the top-right yellow triangles):
...
- 201
2
votes
0
answers
67
views
Learning Manifolds using Gradient Descent
I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight ...
- 325
2
votes
1
answer
43
views
Is it necessary to constrain the size of the neighborhood in LLE to be less than the space dimensionality?
The wikipedia entry on Locally Linear Embedding (LLE) says that LLE can be broken into stages, the first of which is to learn a barycentric linear model of the data with its $k$-nearest neighbors:
$...
- 2,217
2
votes
0
answers
41
views
How to identify manifolds for an optimisation problem
I don't have much experience in topology, but I am interested to know if:
• Given a particular problem and associated cost function, how would one deduce what kind of manifold this problem lies on.
...
- 455
0
votes
0
answers
104
views
Topological approach to create a space between clouds
I have a dataset associated with labels. According to https://arxiv.org/pdf/1802.03426.pdf --> UMAP (Uniform Manifold Approximation and Projection) which is a novel manifold learning technique for ...
- 11
2
votes
1
answer
48
views
Is low rank finite-iteration manifold identification possible?
In sparse optimization, I am trying to solve the problem
$$
\min_{x\in \mathbb R^{n}} \quad f(x) + \|x\|_1
$$
and at optimality, $x^*$ may be sparse. If I define the sparse manifold as
$\mathcal M = ...
- 1,277
2
votes
1
answer
102
views
Residual learning approach + manifold
From: Wikipedia on Residual learning in ANNs
"The intuition on why this works is that the neural network collapses into fewer layers in the initial phase, which makes it easier to learn, and thus ...
- 782
5
votes
2
answers
2k
views
What exactly is called "embedding" in dimensionality reduction?
In the following slide I do not understand the definition of the term embedding.
In the third paragraph, it says it is a mapping from low-dim. to high-dim, but in the last paragraph it suggests that ...
- 1,047
1
vote
0
answers
133
views
Is it possible to compute residual variances of Laplacian Eigenmaps, Diffusion Map and t-SNE to have a scree plots as in PCA?
I will like to do comparative analysis of PCA vs. Laplacian Eigenmaps, Diffusion Map, and <...
- 173
0
votes
0
answers
774
views
connection between residual variance and explained variance PCA
The publication introducing Isomap compares PCA and Isomap by means of
$$\text{residual variance} = 1 - R^2(\hat D_M, D_y)$$
where $R$ is the standard linear correlation coefficient over all ...
- 131
2
votes
1
answer
221
views
What is meant by: neighbor identities are preserved in t-SNE?
From the article on Stochastic Neighbor Embedding (by Geoffrey Hinton and Sam Roweis):
Stochastic Neighbor Embedding (SNE) tries to place
the objects in a low-dimensional space so as to ...
- 133
3
votes
1
answer
605
views
How to determine number of components to retain in manifold learning?
With PCA, one can use the explained variance ratio and keep the number of components that explain 95% of the dataset.
How does one do the same for manifold learning methods for dimensionality ...
2
votes
1
answer
2k
views
t-SNE, PCA and another technique - which one? [closed]
I am comparing dimension reduction techniques and I am utilizing them for data visualizations onto a plane – projections in 2D space.
The input into a projection/dimension reduction techniques is a ...
- 31
2
votes
1
answer
928
views
Why do I get weird results when using high perpexity in t-SNE?
I played around with the t-SNE implementation in scikit-learn and found that increasing perplexity seemed to always result in a torus/circle. I couldn't find any ...
- 145
1
vote
0
answers
226
views
how to check the stability of t-SNE?
I am looking some technique that help us to test the stability of t-SNE for different times running.
- 119
0
votes
0
answers
63
views
why we need to select the smallest singular values in locally linear embedding (LLE)?
I'm learning about locally linear embedding. The cost function for finding embedded data is given by
$\Phi(X) = \Sigma_{ij}M_{ij}(X_i.X_j^T$)
Why we need to select the $2^{nd}$ to $(P+1)^{th}$ ...
- 233
7
votes
1
answer
3k
views
How to know when to use linear dimensionality reduction vs non-linear dimensionality reduction?
I am trying to decide whether to use linear dimensionality reduction methods (eg. PCA) vs. non-linear dimensionality reduction methods (eg. t-SNE) for my high-dimensional data set. However, I know ...
- 71