Questions tagged [manifold-learning]

Manifold learning subsumes techniques conceived for problems where data of interest are assumed to lie on an embedded non-linear manifold within a higher-dimensional space.

Filter by
Sorted by
Tagged with
2 votes
1 answer
14 views

Determine if high dimensional data is multimodal

I have p-dimensional data and I need to determine if that data has significant modes or if it’s clustered in any way. Here p=50, (dense embedding), we have n samples and p <<< n. What are ...
user avatar
  • 21
2 votes
1 answer
56 views

Why does data get so tangled up in high dimension?

When I look at textbooks on classification and machine learning, many of the examples focus on data that is often twisted up such as to avoid linear separation. I have an example picture below. The ...
user avatar
  • 1,150
0 votes
0 answers
15 views

Reproducing kernel hilbert space norm as smoothness functional

Let $K:X \times X \rightarrow \mathbb{R}$ be a Mercer kernel with an associated RKHS $H$ then the norm $|f|_H^2$ can be used as a way to ensure that $f$ is smooth in $H$. If i understand correctly, ...
user avatar
2 votes
0 answers
33 views

Any theory on whether good choices of $k$ depend on $N$ and $D$ in KNN classification?

I am well aware that cross validation is a usual method for selecting hyperparameters. However, I am looking for theoretical guidance on how to pick $k$, the number of neighbors, for a $k$-nearest-...
user avatar
  • 579
0 votes
1 answer
38 views

Variance used in t-SNE

I was going through the t-SNE, but I am bit confused. While the original paper of t-SNE is based on the SNE and SNE uses $\sigma_i^2$ (note the subscript $i$) while calculating the similarity of point ...
user avatar
1 vote
0 answers
20 views

Analytical tools to analyze the characteristics of a data manifold

In the paper "Emergence of separable manifolds in deep language representations," the authors use an analytical tool called Mean Field Theoretic Manifold Analysis to measure the manifold ...
user avatar
  • 11
11 votes
4 answers
1k views

Dimension reduction using space filling curve to avoid "Curse of dimensionality"?

In machine learning, we want to train a model. While training, if the dimension of data is high, we have a problem (Curse of Dimensionality), so we want to reduce the dimension of our data. Since we ...
user avatar
0 votes
0 answers
18 views

spherical embedding of image data

Embedding positive data images onto a spherical manifold using the spherical embedding method doesn't guarantee the generation of positive directional quantities as the embedding positions are ...
user avatar
1 vote
2 answers
415 views

What validation if KFold scores differ a lot? Repeated KFold, LOO or Holdout?

Suppose you are given a medium-sized dataset and you did a KFold validation once. You notice that scores on each old differ noticeably. Which validation type is the most practical? I thought about ...
user avatar
2 votes
0 answers
26 views

State-of-the-art methods for out-of-sample-extension

I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis. I've been using the Nystrom method for this task, and some ...
user avatar
  • 729
0 votes
1 answer
29 views

Simulate data that rotate on itself like DNA

I'm doing an exercise on the reduction of nonlinear dimension in manifold. I want to use LLE for that. And I'd like to simulate data in the form of DNA, so it rotates on itself. That is, a flat ...
user avatar
  • 121
1 vote
0 answers
72 views

how do i calculate (and Apply) Loss gradients with respect to the input (not the weights) of a CNN?

I have a trained generator, i would like to apply a loss function to the output and optimize the input (latent vector) using a gradient decent optimizer. i don't know how to calculate the gradients ...
user avatar
  • 11
1 vote
0 answers
78 views

Manifold Hypothesis vs. Latent Variables Assumption [closed]

As I understand: The manifold hypothesis claims that real world data, although represented in high dimension space, actually lies on a manifold in that space. I.e. that the actual data structure is of ...
user avatar
0 votes
1 answer
354 views

Multivariate Jensen-Shannon divergence

This paper says multivariate Jensen-Shannon divergence is $$JS(\mathbf{p}_1,\dots,\mathbf{p}_K) = \frac{1}{m} \sum KL(\mathbf{p}_i || \bar{\mathbf{p}})$$ with $KL$ being the KL-divergence of the ...
user avatar
  • 3,089
0 votes
0 answers
382 views

Differentiable PCA?

Is there a differentiable method for dimensionality reduction that is either based on PCA or has the properties of: Mathematically or algorithmically defined, e.g. not trained like an ML model or t-...
user avatar
  • 181
2 votes
0 answers
52 views

PCA for non linear manifolds - Yann Lecun, Deep Learning Course, Question from a Lecture

I was watching the following lecture and at the very end of it, one of the students asked LeCun about using PCA for expression and pose feature extraction. https://www.youtube.com/watch?v=0bMe_vCZo30&...
user avatar
1 vote
1 answer
246 views

Testing of hypothesis for the linearity of a data? PCA suggested, but how do we design a statistical test using it? [closed]

Suppose we're given the data set $\{x_1 \dots x_n\}$ in $\mathbb{R}^D$ the $D$-dimensional Euclidean space, and assume this data has intrinsic dimension $d < D.$ N.B. this just means that data is ...
user avatar
  • 681
3 votes
1 answer
846 views

Meaning of "Manifold of interest"

While reading a paper, I stumble upon the following sentence and couldn't figure out its meaning: Informally, for an input set of real images, we say that the set of layer activations (for any ...
user avatar
2 votes
0 answers
29 views

Theoretical justification behind assuming that the data is locally uniformly distributed, as seem to be used by manifold learning community

In at least three or more papers I've been studying that introduced novel algorithms for the estimation of intrinsic dimensionality (ID) based on nearest neighborhood (NN) techniques, I observed that ...
user avatar
  • 681
1 vote
0 answers
15 views

Is there any manifold learning literature where the both the ambient dimension and intrinsic dimension of the data are high?

I'm new to manifold learning, and from my understanding it normally denotes a subject which starts with the assumption that the data $\{x_1 \dots x_n\} \subset (M,g) \subset \mathbb{R}^p,$ where $(M,g)...
user avatar
  • 681
0 votes
0 answers
26 views

When is the manifold hypothesis violated?

If the high-dimensional dataset is of full rank, like an N samples by N features matrix, will the manifold hypothesis be violated? i.e. we cannot assume those data live on a lower-dimensional space?
user avatar
2 votes
0 answers
234 views

Does classic MultiDimensional Scaling reconstructs data upto a rotation when there's no noise and embedding dimensions equals original data dimension?

In the setup of classical MultiDimensional Scaling (MDS), assume that $D:=[d_{ij}]$ be an $n \times n$ distance matrix, i.e. $d(i,i)=0, d(i,j)=d(j,i) > 0 \forall i, j = 1 \dots n.$ Assume that: ...
user avatar
  • 681
4 votes
0 answers
96 views

In Multidimensional Scaling (MDS), is it safe to assume that the optimal embedding dimension grows with the growth of sample size?

My question is more of a theoretical nature, so it'd be great to have some references to papers, but it'd be also nice to see some experiments. Let $D:=[d_{ij}]$ be an $n \times n$ distance matrix, i....
user avatar
  • 681
1 vote
0 answers
52 views

How does high dimensional data residing in low dimensional manifold have high variance?

I was pondering over the fact that for a very high dimensional data, if it lies in a low dimesional manifold, then in the bias variance decomposition for it, the variance will be high and bias will be ...
user avatar
1 vote
0 answers
28 views

Why RANDOM noise images always predicted as BIRD?

Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%. However when feeding into 5000 random noise images (Gaussian noise with the ...
user avatar
0 votes
0 answers
43 views

Manifold optimization theory

Does there exist some elementary book for optimization on manifolds for problems in finance/economics?
0 votes
1 answer
132 views

Spectral embedding: interpretation of new dimensions

I'm trying to gain an intuition for the 2nd dimension in the spectral embedding of an S-shaped dataset as in this example: The 1st dimension seems to neatly capture the local similarity between ...
user avatar
  • 101
0 votes
1 answer
40 views

Linear and Non-Linear dimensionality reduction with missing variables

I'm trying to compare the two types of reduction by applying it to a list of ingredients for about 1000 sponge cakes. The ingredient lists do however have miss certain ingredients out for some cakes, ...
user avatar
2 votes
1 answer
494 views

Dimensionality Reduction for Optimally Preserving KNN

Do any dimensionality reduction techniques find embeddings which optimally preserve the K-nearest neighbors of each point? If no algorithm provably does this, are there algorithms which heuristically ...
user avatar
4 votes
1 answer
85 views

Quantifying a manifold folding unto itself

I have a dataset of ~7k scattered points in 3D which represents a manifold that may or may not "fold unto itself". Here's an example where this does happen (look at the top-right yellow triangles): ...
user avatar
  • 181
2 votes
0 answers
49 views

Learning Manifolds using Gradient Descent

I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight ...
user avatar
  • 265
2 votes
1 answer
39 views

Is it necessary to constrain the size of the neighborhood in LLE to be less than the space dimensionality?

The wikipedia entry on Locally Linear Embedding (LLE) says that LLE can be broken into stages, the first of which is to learn a barycentric linear model of the data with its $k$-nearest neighbors: $...
user avatar
  • 2,047
2 votes
0 answers
31 views

How to identify manifolds for an optimisation problem

I don't have much experience in topology, but I am interested to know if: • Given a particular problem and associated cost function, how would one deduce what kind of manifold this problem lies on. ...
user avatar
0 votes
0 answers
100 views

Topological approach to create a space between clouds

I have a dataset associated with labels. According to https://arxiv.org/pdf/1802.03426.pdf --> UMAP (Uniform Manifold Approximation and Projection) which is a novel manifold learning technique for ...
user avatar
2 votes
1 answer
44 views

Is low rank finite-iteration manifold identification possible?

In sparse optimization, I am trying to solve the problem $$ \min_{x\in \mathbb R^{n}} \quad f(x) + \|x\|_1 $$ and at optimality, $x^*$ may be sparse. If I define the sparse manifold as $\mathcal M = ...
user avatar
  • 1,247
2 votes
1 answer
89 views

Residual learning approach + manifold

From: Wikipedia on Residual learning in ANNs "The intuition on why this works is that the neural network collapses into fewer layers in the initial phase, which makes it easier to learn, and thus ...
user avatar
4 votes
2 answers
1k views

What exactly is called "embedding" in dimensionality reduction?

In the following slide I do not understand the definition of the term embedding. In the third paragraph, it says it is a mapping from low-dim. to high-dim, but in the last paragraph it suggests that ...
user avatar
  • 937
1 vote
0 answers
105 views

Is it possible to compute residual variances of Laplacian Eigenmaps, Diffusion Map and t-SNE to have a scree plots as in PCA?

I will like to do comparative analysis of PCA vs. Laplacian Eigenmaps, Diffusion Map, and <...
user avatar
0 votes
0 answers
619 views

connection between residual variance and explained variance PCA

The publication introducing Isomap compares PCA and Isomap by means of $$\text{residual variance} = 1 - R^2(\hat D_M, D_y)$$ where $R$ is the standard linear correlation coefficient over all ...
user avatar
  • 131
2 votes
1 answer
159 views

What is meant by: neighbor identities are preserved in t-SNE?

From the article on Stochastic Neighbor Embedding (by Geoffrey Hinton and Sam Roweis): Stochastic Neighbor Embedding (SNE) tries to place the objects in a low-dimensional space so as to ...
user avatar
2 votes
1 answer
455 views

How to determine number of components to retain in manifold learning?

With PCA, one can use the explained variance ratio and keep the number of components that explain 95% of the dataset. How does one do the same for manifold learning methods for dimensionality ...
user avatar
2 votes
1 answer
2k views

t-SNE, PCA and another technique - which one? [closed]

I am comparing dimension reduction techniques and I am utilizing them for data visualizations onto a plane – projections in 2D space. The input into a projection/dimension reduction techniques is a ...
user avatar
  • 21
2 votes
1 answer
745 views

Why do I get weird results when using high perpexity in t-SNE?

I played around with the t-SNE implementation in scikit-learn and found that increasing perplexity seemed to always result in a torus/circle. I couldn't find any ...
user avatar
1 vote
0 answers
181 views

how to check the stability of t-SNE?

I am looking some technique that help us to test the stability of t-SNE for different times running.
user avatar
0 votes
0 answers
52 views

why we need to select the smallest singular values in locally linear embedding (LLE)?

I'm learning about locally linear embedding. The cost function for finding embedded data is given by $\Phi(X) = \Sigma_{ij}M_{ij}(X_i.X_j^T$) Why we need to select the $2^{nd}$ to $(P+1)^{th}$ ...
user avatar
  • 203
6 votes
1 answer
2k views

How to know when to use linear dimensionality reduction vs non-linear dimensionality reduction?

I am trying to decide whether to use linear dimensionality reduction methods (eg. PCA) vs. non-linear dimensionality reduction methods (eg. t-SNE) for my high-dimensional data set. However, I know ...
user avatar
  • 61
1 vote
0 answers
181 views

Nonlinear dimensionality reduction in Java/on JVM

Is there a (decently) fast implementation of Manifold Learning algorithms on JVM? I tried Smile library but it takes dozen minutes to run its Isomap on my computer, while scikit-learn ...
user avatar
3 votes
0 answers
292 views

Topological data analysis and evaluating dimensionality reduction

I did an exploration some time ago on using TDA tools to see how topological features change after application of some nonlinear dimensionality reduction methods. For example I found out that, for ...
user avatar
2 votes
1 answer
55 views

Help in understanding a formula used in dimension estimation from nearest neighbor distances

In this paper, Abstract—The intrinsic dimensionality of a set of patterns is important in determining an appropriate number of features for representing the data and whether a reasonable two- or ...
user avatar
  • 925
1 vote
1 answer
176 views

Diffusion Maps implementation

Suppose I have n data points lying in $\mathbf{R}^3$. Then, after defining my Gaussian diffusion kernel $k(x,y)$ and computing matrix $K$, I obtain $P$, whose entries $P_{ij}=p(x_i,y_i)$ is the ...
user avatar