Skip to main content

Questions tagged [manifold-learning]

Manifold learning subsumes techniques conceived for problems where data of interest are assumed to lie on an embedded non-linear manifold within a higher-dimensional space.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

How to interpret Diffusion Maps for the iris dataset?

This might be a poor exercise but I'm trying to understand the methods of paper and if it makes sense to adapt my linear-based workflow with PCA to non-linear manifold methods; thought trying out ...
O.rka's user avatar
  • 1,472
0 votes
0 answers
43 views

Why do we expect the eigenvalues of the Gramian matrix found by Maximum Variance Unfolding to capture the number of degrees of freedom in the data?

Maximum Variance Unfolding (MVU) is a manifold learning method which, like other forms of dimensionality reduction, makes the assumption that whatever (high-dimensional) data we're dealing with "...
Saucy Goat's user avatar
1 vote
0 answers
80 views

Is it correct to do SVD from the latent space of an autoencoder?

Is it correct to do SVD from the latent space of an autoencoder? I am asking because I think that by performing SVD from a latent space, and plotting the singular values, it is possible to know the ...
Carlos's user avatar
  • 11
4 votes
0 answers
345 views

Explanation of UMAP assumptions

Uniform Manifold Approximation (UMAP) is a technique for dimensionality reduction and visualization. The author of UMAP states that the algorithm is founded on three assumptions about the data: The ...
dariober's user avatar
  • 4,754
3 votes
2 answers
328 views

What is the relationship between noise reduction and dimension reduction?

My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data ...
Douw Marx's user avatar
2 votes
1 answer
104 views

Determine if high dimensional data is multimodal

I have p-dimensional data and I need to determine if that data has significant modes or if it’s clustered in any way. Here p=50, (dense embedding), we have n samples and p <<< n. What are ...
Martin's user avatar
  • 21
2 votes
1 answer
111 views

Why does data get so tangled up in high dimension?

When I look at textbooks on classification and machine learning, many of the examples focus on data that is often twisted up such as to avoid linear separation. I have an example picture below. The ...
krishnab's user avatar
  • 1,522
0 votes
0 answers
204 views

Reproducing kernel hilbert space norm as smoothness functional

Let $K:X \times X \rightarrow \mathbb{R}$ be a Mercer kernel with an associated RKHS $H$ then the norm $|f|_H^2$ can be used as a way to ensure that $f$ is smooth in $H$. If i understand correctly, ...
endeavor's user avatar
  • 183
2 votes
0 answers
35 views

Any theory on whether good choices of $k$ depend on $N$ and $D$ in KNN classification?

I am well aware that cross validation is a usual method for selecting hyperparameters. However, I am looking for theoretical guidance on how to pick $k$, the number of neighbors, for a $k$-nearest-...
ted's user avatar
  • 741
1 vote
1 answer
791 views

Variance used in t-SNE

While the original paper of t-SNE is based on the SNE and SNE uses $\sigma_i^2$ (note the subscript $i$) while calculating the similarity of point $x_i$ WRT other points $j$. i.e., for each data point ...
Aarif Rather's user avatar
1 vote
0 answers
30 views

Analytical tools to analyze the characteristics of a data manifold

In the paper "Emergence of separable manifolds in deep language representations," the authors use an analytical tool called Mean Field Theoretic Manifold Analysis to measure the manifold ...
engnad's user avatar
  • 11
10 votes
4 answers
1k views

Dimension reduction using space filling curve to avoid "Curse of dimensionality"?

In machine learning, we want to train a model. While training, if the dimension of data is high, we have a problem (Curse of Dimensionality), so we want to reduce the dimension of our data. Since we ...
Shriman Keshri's user avatar
1 vote
2 answers
761 views

What validation if KFold scores differ a lot? Repeated KFold, LOO or Holdout?

Suppose you are given a medium-sized dataset and you did a KFold validation once. You notice that scores on each old differ noticeably. Which validation type is the most practical? I thought about ...
Revolucion for Monica's user avatar
2 votes
0 answers
62 views

State-of-the-art methods for out-of-sample-extension

I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis. I've been using the Nystrom method for this task, and some ...
Roy's user avatar
  • 839
0 votes
1 answer
32 views

Simulate data that rotate on itself like DNA

I'm doing an exercise on the reduction of nonlinear dimension in manifold. I want to use LLE for that. And I'd like to simulate data in the form of DNA, so it rotates on itself. That is, a flat ...
Marc's user avatar
  • 121
1 vote
0 answers
173 views

how do i calculate (and Apply) Loss gradients with respect to the input (not the weights) of a CNN?

I have a trained generator, i would like to apply a loss function to the output and optimize the input (latent vector) using a gradient decent optimizer. i don't know how to calculate the gradients ...
Rooz's user avatar
  • 11
1 vote
0 answers
177 views

Manifold Hypothesis vs. Latent Variables Assumption [closed]

As I understand: The manifold hypothesis claims that real world data, although represented in high dimension space, actually lies on a manifold in that space. I.e. that the actual data structure is of ...
Maverick Meerkat's user avatar
0 votes
1 answer
1k views

Multivariate Jensen-Shannon divergence

This paper says multivariate Jensen-Shannon divergence is $$JS(\mathbf{p}_1,\dots,\mathbf{p}_K) = \frac{1}{m} \sum KL(\mathbf{p}_i || \bar{\mathbf{p}})$$ with $KL$ being the KL-divergence of the ...
develarist's user avatar
  • 4,015
1 vote
0 answers
717 views

Differentiable PCA? [closed]

Is there a differentiable method for dimensionality reduction that is either based on PCA or has the properties of: Mathematically or algorithmically defined, e.g. not trained like an ML model or t-...
Yaoshiang's user avatar
  • 271
2 votes
0 answers
68 views

PCA for non linear manifolds - Yann Lecun, Deep Learning Course, Question from a Lecture

I was watching the following lecture and at the very end of it, one of the students asked LeCun about using PCA for expression and pose feature extraction. https://www.youtube.com/watch?v=0bMe_vCZo30&...
Anton Nemchenko's user avatar
1 vote
1 answer
431 views

Testing of hypothesis for the linearity of a data? PCA suggested, but how do we design a statistical test using it? [closed]

Suppose we're given the data set $\{x_1 \dots x_n\}$ in $\mathbb{R}^D$ the $D$-dimensional Euclidean space, and assume this data has intrinsic dimension $d < D.$ N.B. this just means that data is ...
Mathmath's user avatar
  • 761
4 votes
1 answer
1k views

Meaning of "Manifold of interest"

While reading a paper, I stumble upon the following sentence and couldn't figure out its meaning: Informally, for an input set of real images, we say that the set of layer activations (for any ...
Ferdinand Mom's user avatar
2 votes
0 answers
38 views

Theoretical justification behind assuming that the data is locally uniformly distributed, as seem to be used by manifold learning community

In at least three or more papers I've been studying that introduced novel algorithms for the estimation of intrinsic dimensionality (ID) based on nearest neighborhood (NN) techniques, I observed that ...
Mathmath's user avatar
  • 761
1 vote
0 answers
22 views

Is there any manifold learning literature where the both the ambient dimension and intrinsic dimension of the data are high?

I'm new to manifold learning, and from my understanding it normally denotes a subject which starts with the assumption that the data $\{x_1 \dots x_n\} \subset (M,g) \subset \mathbb{R}^p,$ where $(M,g)...
Mathmath's user avatar
  • 761
0 votes
0 answers
67 views

When is the manifold hypothesis violated?

If the high-dimensional dataset is of full rank, like an N samples by N features matrix, will the manifold hypothesis be violated? i.e. we cannot assume those data live on a lower-dimensional space?
TracyYXChen's user avatar
2 votes
0 answers
318 views

Does classic MultiDimensional Scaling reconstructs data upto a rotation when there's no noise and embedding dimensions equals original data dimension?

In the setup of classical MultiDimensional Scaling (MDS), assume that $D:=[d_{ij}]$ be an $n \times n$ distance matrix, i.e. $d(i,i)=0, d(i,j)=d(j,i) > 0 \forall i, j = 1 \dots n.$ Assume that: ...
Mathmath's user avatar
  • 761
4 votes
0 answers
123 views

In Multidimensional Scaling (MDS), is it safe to assume that the optimal embedding dimension grows with the growth of sample size?

My question is more of a theoretical nature, so it'd be great to have some references to papers, but it'd be also nice to see some experiments. Let $D:=[d_{ij}]$ be an $n \times n$ distance matrix, i....
Mathmath's user avatar
  • 761
1 vote
0 answers
100 views

How does high dimensional data residing in low dimensional manifold have high variance?

I was pondering over the fact that for a very high dimensional data, if it lies in a low dimesional manifold, then in the bias variance decomposition for it, the variance will be high and bias will be ...
Anshul Yadav's user avatar
1 vote
0 answers
35 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
dmrak's user avatar
  • 11
0 votes
0 answers
48 views

Manifold optimization theory

Does there exist some elementary book for optimization on manifolds for problems in finance/economics?
0 votes
1 answer
225 views

Spectral embedding: interpretation of new dimensions

I'm trying to gain an intuition for the 2nd dimension in the spectral embedding of an S-shaped dataset as in this example: The 1st dimension seems to neatly capture the local similarity between ...
PFD's user avatar
  • 101
0 votes
1 answer
48 views

Linear and Non-Linear dimensionality reduction with missing variables

I'm trying to compare the two types of reduction by applying it to a list of ingredients for about 1000 sponge cakes. The ingredient lists do however have miss certain ingredients out for some cakes, ...
Dan Savage's user avatar
2 votes
1 answer
625 views

Dimensionality Reduction for Optimally Preserving KNN

Do any dimensionality reduction techniques find embeddings which optimally preserve the K-nearest neighbors of each point? If no algorithm provably does this, are there algorithms which heuristically ...
user2663116's user avatar
4 votes
1 answer
136 views

Quantifying a manifold folding unto itself

I have a dataset of ~7k scattered points in 3D which represents a manifold that may or may not "fold unto itself". Here's an example where this does happen (look at the top-right yellow triangles): ...
Dev-iL's user avatar
  • 201
2 votes
0 answers
70 views

Learning Manifolds using Gradient Descent

I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight ...
Alfred's user avatar
  • 325
2 votes
1 answer
47 views

Is it necessary to constrain the size of the neighborhood in LLE to be less than the space dimensionality?

The wikipedia entry on Locally Linear Embedding (LLE) says that LLE can be broken into stages, the first of which is to learn a barycentric linear model of the data with its $k$-nearest neighbors: $...
Him's user avatar
  • 2,267
2 votes
0 answers
41 views

How to identify manifolds for an optimisation problem

I don't have much experience in topology, but I am interested to know if: • Given a particular problem and associated cost function, how would one deduce what kind of manifold this problem lies on. ...
tisPrimeTime's user avatar
0 votes
0 answers
104 views

Topological approach to create a space between clouds

I have a dataset associated with labels. According to https://arxiv.org/pdf/1802.03426.pdf --> UMAP (Uniform Manifold Approximation and Projection) which is a novel manifold learning technique for ...
davegaut's user avatar
2 votes
1 answer
48 views

Is low rank finite-iteration manifold identification possible?

In sparse optimization, I am trying to solve the problem $$ \min_{x\in \mathbb R^{n}} \quad f(x) + \|x\|_1 $$ and at optimality, $x^*$ may be sparse. If I define the sparse manifold as $\mathcal M = ...
Y. S.'s user avatar
  • 1,277
2 votes
1 answer
105 views

Residual learning approach + manifold

From: Wikipedia on Residual learning in ANNs "The intuition on why this works is that the neural network collapses into fewer layers in the initial phase, which makes it easier to learn, and thus ...
Rahul Deora's user avatar
5 votes
2 answers
2k views

What exactly is called "embedding" in dimensionality reduction?

In the following slide I do not understand the definition of the term embedding. In the third paragraph, it says it is a mapping from low-dim. to high-dim, but in the last paragraph it suggests that ...
user_anon's user avatar
  • 1,047
1 vote
0 answers
149 views

Is it possible to compute residual variances of Laplacian Eigenmaps, Diffusion Map and t-SNE to have a scree plots as in PCA?

I will like to do comparative analysis of PCA vs. Laplacian Eigenmaps, Diffusion Map, and <...
Hello World's user avatar
1 vote
0 answers
869 views

connection between residual variance and explained variance PCA

The publication introducing Isomap compares PCA and Isomap by means of $$\text{residual variance} = 1 - R^2(\hat D_M, D_y)$$ where $R$ is the standard linear correlation coefficient over all ...
gdkrmr's user avatar
  • 141
2 votes
1 answer
238 views

What is meant by: neighbor identities are preserved in t-SNE?

From the article on Stochastic Neighbor Embedding (by Geoffrey Hinton and Sam Roweis): Stochastic Neighbor Embedding (SNE) tries to place the objects in a low-dimensional space so as to ...
Nicolai Iversen's user avatar
3 votes
1 answer
771 views

How to determine number of components to retain in manifold learning?

With PCA, one can use the explained variance ratio and keep the number of components that explain 95% of the dataset. How does one do the same for manifold learning methods for dimensionality ...
Mohammad Riazi's user avatar
2 votes
1 answer
3k views

t-SNE, PCA and another technique - which one? [closed]

I am comparing dimension reduction techniques and I am utilizing them for data visualizations onto a plane – projections in 2D space. The input into a projection/dimension reduction techniques is a ...
xdaniel's user avatar
  • 31
2 votes
1 answer
1k views

Why do I get weird results when using high perpexity in t-SNE?

I played around with the t-SNE implementation in scikit-learn and found that increasing perplexity seemed to always result in a torus/circle. I couldn't find any ...
Mathias Andersen's user avatar
1 vote
0 answers
243 views

how to check the stability of t-SNE?

I am looking some technique that help us to test the stability of t-SNE for different times running.
user8523104's user avatar
0 votes
0 answers
69 views

why we need to select the smallest singular values in locally linear embedding (LLE)?

I'm learning about locally linear embedding. The cost function for finding embedded data is given by $\Phi(X) = \Sigma_{ij}M_{ij}(X_i.X_j^T$) Why we need to select the $2^{nd}$ to $(P+1)^{th}$ ...
pauli's user avatar
  • 253
7 votes
1 answer
3k views

How to know when to use linear dimensionality reduction vs non-linear dimensionality reduction?

I am trying to decide whether to use linear dimensionality reduction methods (eg. PCA) vs. non-linear dimensionality reduction methods (eg. t-SNE) for my high-dimensional data set. However, I know ...
M. T.'s user avatar
  • 71