Questions tagged [dimensionality-reduction]

Refers to techniques for reducing a large number of variables or dimensions spanned by data to a smaller number of dimensions while preserving as much information about the data as possible. Prominent methods include PCA, MDS, Isomap, etc. The two main subclasses of techniques: feature extraction and feature selection.

303 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
509 views

How can an autoencoder unroll the swiss roll?

I'm trying to unroll the swiss roll (from 3D to 2D) using an autoencoder, but it keeps getting stuck in local optima: the swiss roll ends up squashed rather than unrolled. It's no better than using ...
4
votes
0answers
70 views

Nystrom approximation with inexact/stochastic kernel evaluation

Suppose we have several data points $x_1,\ldots,x_m\in\mathbb R^n$ as well as a positive definite kernel $K(x,y):\mathbb R^n\times\mathbb R^n\to\mathbb R$ that can be written in the form $$K(x,y)=\...
4
votes
0answers
283 views

Intuition behind perplexity parameter in t-SNE

While reading Laurens van der Maaten's paper about t-SNE we can encounter the following statement about perplexity: The perplexity can be interpreted as a smooth measure of the effective number of ...
4
votes
0answers
331 views

Differences between t-SNE and SOM

I have some high dimensional data and I want to reduce it to 2 dimensions for visualization. The goal is to color the points in this 2D space to see whether there is any clustering due to different ...
4
votes
0answers
170 views

Why are the discriminant axes in linear discriminant analysis (LDA) not orthogonal?

This may be a quite silly question and please correct me if I'm wrong. The discriminants (discriminant axes) are essentially eigenvectors of $\mathrm{Cov}_\mathrm{within}^{-1} \mathrm{Cov}_\mathrm{...
4
votes
0answers
435 views

How does t-SNE slow down with increasing number of dimensions?

I'm trying to understand the computational bounds of t-SNE. It's learned with SGD, so it'll have to go through some number of gradient-descent iterations. We can ignore that here, and focus on the ...
4
votes
1answer
363 views

Does rank of observation matrix tell anything useful when applying machine learning?

Suppose I have an observation matrix of size $N \times M$ where $N$ is the number of samples and $M$ is the number of variables. If the rank of the observation matrix is $R<M$, does it tell ...
4
votes
0answers
575 views

Dimensionality reduction with locality-sensitive hashing

I have a conceptual understanding of locality-sensitive hashing (LSH) in relation to near-neighbour search. However, the articles that I have read so far seem to gloss over the details of how LSH can ...
4
votes
0answers
364 views

Comparing principal components if number of variables changes

I have a database that contains daily stock returns of more than a thousand stocks for many decades. I would like to achieve the following goals: Construct a time-varying measure of co-movement (or ...
4
votes
0answers
2k views

Why would concatenating feature vectors lead to better estimates?

I wish to estimate the state of a system from two separate and disparate observations. A simple approach that I have seen in some literature is to combine the feature vectors (observations) by simply ...
3
votes
1answer
112 views

How can Factor Analysis be used to remove questions from a survey?

Suppose I have a psychological questionnaire asking 30 questions about a person's mental health (on a Likert-scale 1-7). These 30 questions fall into 7 separate, but correlated categories. The ...
3
votes
0answers
942 views

Is MCA equivalent to PCA when all variables are binary?

I am looking to apply principal component analysis on binary (true/false) data, and I have come across the "equivalence between PCA and MCA" (Multiple Correspondense Analysis) for binary data, but ...
3
votes
1answer
84 views

How to analyze this biplot of PCA?

I have dones a PCA analysis about measurement of a fish morphometric between female and male. After the PCA result came out with biplot graph, I was a little bit confused to interpret this data. It ...
3
votes
0answers
26 views

Reducing a logistic model used for prediction

I'm developing a logistic regression used for prediction. I have pre-selected, based on prev. literature, 15 candidate predictors (fitting my ~200 events). Now, I want a reduced/more parsimonious ...
3
votes
0answers
234 views

General PCA optimization problem

I was looking at the PCA optimization problem, which is finding a matrix $U \in \mathbb{R}^{d\times n}$, $n \le d$, that solves the problem $$\max{tr(U^TCU)},\ \ \ s.t. U^TU = I, $$ where $C$ is the ...
3
votes
0answers
133 views

Which dimensionality reduction technique preserves the k nearest neighbors (euclidean space)?

I'm looking for a lower dimensional projection of data such that the k nearest neighbors (in Euclidean space) in high dimensions remain the k nearest neighbors in low dimensions. I found that Isomap ...
3
votes
0answers
218 views

Topological data analysis and evaluating dimensionality reduction

I did an exploration some time ago on using TDA tools to see how topological features change after application of some nonlinear dimensionality reduction methods. For example I found out that, for ...
3
votes
0answers
108 views

Walkthrough for Locally Linear Embedding

Can someone please walk me through the final step for LLE? Specifically, computing the coordinates of the vectors $Y_i$ on the lower dimensional manifold. Disclaimer: I am aware of another post ...
3
votes
0answers
88 views

PCA with one “known” component

(The title might not be very clear, any recommendation welcome) I have a $1 + n$ dimensional dataset. The first dimension measures a specific concept I am interested in and the $n$ remaining ...
3
votes
0answers
1k views

Autoencoder with Mixed Data

Is it reasonable? The categorical features can be binary ("true" or "false") or strings, which are one-hot encoded. Some continuous features may be integers, which are treated as real values. If an ...
3
votes
0answers
128 views

Inference on Treatment Effects after Selection among High-Dimensional Controls

I am reading the work by Belloni et al (2014), see the name in the title (weblink here). What I understand is that they assume that the outcome can be well approximated by a small/sparse set of ...
3
votes
0answers
199 views

How does PCA maximise Total Variance without maximising Co-variance?

https://stats.stackexchange.com/a/3374/92071 - In PCA, the components are actual orthogonal linear combinations that maximize the total variance. In FA, the factors are linear combinations that ...
3
votes
0answers
115 views

Data reduction and xgboost(or other boosting and decisision tree methods)

I wonder, does data reduction(ex:factor analysis) have an impact on the result of boosting(ex:xgboost) or decision trees methods other than time gain?
3
votes
0answers
540 views

Elastic net regularization - variables penalization

I have a data with ~ 3000 factor predictors with ~ 6 levels, many rows (300k+), and binary Y (trying to predict probability of event). There are many groups of variables that are highly correlated. I ...
3
votes
0answers
302 views

How to deal with different sizes of sentences when giving them as input to a Neural Network?

I am giving a sentence as input to a tree structured Neural Network, where the leaf nodes will be the word vectors of the words in the sentence. That tree will be a binarized constituency(see the ...
3
votes
0answers
42 views

Creating consensus from multiple methods of measuring the same entity with some missing values

Imagine we have C cars and D drivers, and each driver takes a large subset of these C cars in order to test the rate of fuel consumption for some fixed amount of fuel (let's assume that the number of ...
3
votes
0answers
490 views

Variable reduction by means of ANOVA?

I have a typical problem with several variables and a large amount of data which are not important right now. The goal of the study is to relate variable $Y$ with variables $X_1,X_2,...,X_n$. I have ...
3
votes
0answers
130 views

How to reconstruct the original data when using manifold learning?

I'm using Isomap to reduce the dimensionality of my data. Isomap use geodesic distance rather than Euclidean distance to perform a MDS. Now I want to reconstruct my original data with a lower ...
3
votes
0answers
150 views

Is there a principled approach to dimensionality reduction?

Background I have walked around my office environment with a video camera. From these image sequences, I wish to determine whether or not there is a 2D embedding that approximates the camera's ...
3
votes
0answers
72 views

Reducing the dimension of an embedding

Let $O \in \mathbb R^{p\times m}$ be a data matrix of observations. Suppose we are given a model $\mu : \mathbb R^n \rightarrow \mathbb R^m$ which is able to approximately fit the observations. Fix $...
3
votes
0answers
1k views

Factor analysis for categorical target variable

I'm doing some research into factor analysis and I've hit that barrier where I don't know what search terms to use. I'm trying to see if something is possible. Basically I have a data set with ~100 ...
2
votes
0answers
15 views

How is explained variance in sparse PCA calculated?

Sparse PCA is a technique proposed by Zou et all in this paper. In usual PCA the obtained loadings are orthonormal, and the resulting scores are uncorrelated. However, in sparse PCA you give up these ...
2
votes
2answers
28 views

Understand important features in UMAP

I am using a dimensionality reduction algorithm (UMAP) to cluster high-dimensional data. Particularly, I have ~50000 vectors of dimension ~20000 to visualise. These vectors are highly structured: ...
2
votes
0answers
22 views

Clustering of very high dimensional data and large number of examples without losing info in dimensions

I'm trying to get a grasp on scalability of clustering algorithms, and have a toy example in mind. Let's say I have around a million or so songs from $50$ genres. Each song has characteristics - some ...
2
votes
0answers
67 views

Find correlations based on large multivariable time-based data with one output per dataset

I am not well versed in anything beyond basic statistics but have been tasked with coming up with a "grading" scale for wear on a part based on data we have collected. I am in need of help figuring ...
2
votes
0answers
63 views

How to conduct a principal component analysis on data set with large number of zeros

I have data for percentage cover of plant species in 500 sites. There are columns for 30 different species in the data set and I would like to drastically reduce this down to a manageable number of ...
2
votes
0answers
51 views

Principal Components' relation with variables having lower variance

This is a philosophical question about PCA, and not a direct coding question. I understand that PCA is a dimensionality reduction technique which results in a certain set of PCs, each PC being a ...
2
votes
0answers
29 views

If I recover VAR model coefficients from PCA-derived coefficients, do I need to ensure that the model has zero cross-correlation in the residuals?

I am investigating how to appropriately combine PCA with VAR modeling. I am using PCA to reduce the number of vars I fit to a VAR model, and am attempting to recover the non-dim. reduced coefficients ...
2
votes
0answers
16 views

Searching intersection of elements in subsets with approximation

Task: Having a large number of transactions that consists of distinct elements from one large set $S$ I need to find transactions in which items have intersection with more than 20% of items in ...
2
votes
0answers
365 views

Principal components: Can I interpret PCA as essentially a change of basis

I was hoping that someone could simply validate or correct my interpretation of Principal Components Analysis. There are a lot of questions on this site about Principal Components analysis--some ...
2
votes
0answers
30 views

PCA: mean of marginal distribution of high-dimensional vector

Consider the following probabilistic model: $$p(x) = \mathcal{N}(0, I_d), \ x \in \mathbb{R}^d$$ $$p(y|x) = \mathcal{N}(Wx + \mu, \sigma^2I_D), \ y, \mu \in \mathbb{R}^D, W \in \mathbb{R}^{D\times d}...
2
votes
0answers
210 views

What's wrong with my solution to canonical correlation analysis (CCA) using the SVD

I am working through the derivations for solving CCA in A Tutorial on Canonical Correlation Methods. Right now, I am trying to solve CCA using SVD (bottom of page 95:7). For completeness, I include ...
2
votes
0answers
82 views

Recovering a distance matrix from nonnegative sparse correlation matrix?

After doing extensive literature research in all sorts of science I am completely puzzled. I am trying to find out what the state-of-the-art techniques would be to recover a (let's say euclidean) ...
2
votes
0answers
151 views

Relationship between Intrinsic Dimension and Principal Components - PCA

I have a data space with 144 features 12x12 grid : and I know that I can fully describe it in M = 6 features with the help of PCA. My data space has intrinsic dimensionality = 3 (three grades of ...
2
votes
0answers
23 views

Removing the influence of undesirable features in clustering

In some machine learning problem, we have objects represented as feature vectors $X$. We want to cluster them. We want some features of $X$ to have no influence on the clustering. Call $X_1$ such a ...
2
votes
0answers
176 views

Dimmensionality reduction for highly dimensional multivariate time series with few time steps

I am looking for a dimmensionality reduction technique that is ideally compatible with LSTM. The dataset I am working with is a multivariate time series with 10 time points and ~13000 features. The ...
2
votes
1answer
135 views

Dimension in Locally Linear Embedding (LLE)

I am using LLE to do nonlinear dimensionality reduction. In my understanding, in the step 3, the eigendecomposition problem is with respect to the matrix M which has the dimension NxN (N is the number ...
2
votes
0answers
109 views

Dimensionality reduction with discrete comparisons

I have data from many people with the following shape (from a survey): $a$ is more similar to $b$ than $c$ where $a, b, c$ are visual stimuli. I would like to visualize the data in a nice one-...
2
votes
1answer
187 views

Gaussian Process Latent Variable Model Optimisation

I am attempting to implement the Nonlinear Gaussian Process Latent Variable Model, as per Lawrence 2005 and have the gradient with respect to the kernel as follows(Eq 10 in paper): $\frac{\partial L}{...
2
votes
0answers
107 views

For what kind of model do the minimum description length (MDL) and the Bayesian information criterion (BIC) give the same estimation?

It is known that BIC=-MDL in many cases. For what kind of ,model the number of dimension is same and for what kind of model they differ?