Skip to main content
240 votes
Accepted

How to reverse PCA and reconstruct original variables from several principal components?

PCA computes eigenvectors of the covariance matrix ("principal axes") and sorts them by their eigenvalues (amount of explained variance). The centered data can then be projected onto these principal ...
amoeba's user avatar
  • 106k
74 votes
Accepted

What is "reduced-rank regression" all about?

1. What is reduced-rank regression (RRR)? Consider multivariate multiple linear regression, i.e. regression with $p$ independent variables and $q$ dependent variables. Let $\mathbf X$ and $\mathbf Y$ ...
amoeba's user avatar
  • 106k
43 votes

Relationship between SVD and PCA. How to use SVD to perform PCA?

I wrote a Python & Numpy snippet that accompanies @amoeba's answer and I leave it here in case it is useful for someone. The comments are mostly taken from @amoeba's answer. ...
43 votes
Accepted

Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?

The main reason that $t$-SNE is not used in classification models is that it does not learn a function from the original space to the new (lower) dimensional one. As such, when we would try to use our ...
usεr11852's user avatar
  • 44.8k
39 votes
Accepted

Is PCA always recommended?

Blindly using PCA is a recipe for disaster. (As an aside, automatically applying any method is not a good idea, because what works in one context is not guaranteed to work in another. We can formalize ...
Sycorax's user avatar
  • 92.3k
35 votes
Accepted

Should data be centered+scaled before applying t-SNE?

Centering shouldn't matter since the algorithm only operates on distances between points, however rescaling is necessary if you want the different dimensions to be treated with equal importance, since ...
jon_simon's user avatar
  • 2,049
33 votes

Relationship between SVD and PCA. How to use SVD to perform PCA?

Let me start with PCA. Suppose that you have $n$ data points comprised of $d$ numbers (or dimensions) each. If you center this data (subtract the mean data point $\mu$ from each data vector $x_i$) you ...
Andre P's user avatar
  • 511
33 votes
Accepted

Intuitive explanation of how UMAP works, compared to t-SNE

You said that your understanding of t-SNE is based on https://www.youtube.com/watch?v=NEaUSP4YerM and you are looking for an explanation of UMAP on a similar level. I watched this video and it is ...
amoeba's user avatar
  • 106k
32 votes
Accepted

Nystroem Method for Kernel Approximation

Let's derive the Nyström approximation in a way that should make the answers to your questions clearer. The key assumption in Nyström is that the kernel function is of rank $m$. (Really we assume that ...
Danica's user avatar
  • 25.1k
26 votes
Accepted

How can t-SNE or UMAP embed new (test) data, given that they are nonparametric?

Great question. I will answer it using t-SNE because I assume it is familiar to more people. I think UMAP is very promising and is a great contribution but to be honest I am getting a little bit ...
amoeba's user avatar
  • 106k
26 votes

Intuitive explanation of how UMAP works, compared to t-SNE

The main difference between t-SNE and UMAP is the interpretation of the distance between objects or "clusters". I use the quotation marks since both algorithms are not meant for clustering - they are ...
Edgar's user avatar
  • 1,701
25 votes
Accepted

Modelling with more variables than data points

It's certainly possible to fit good models when there are more variables than data points, but this must be done with care. When there are more variables than data points, the problem may not have a ...
user20160's user avatar
  • 32.8k
24 votes
Accepted

What is the meaning of the axes in t-SNE?

Individual axes in t-SNE have no meaning at all. Algorithms such as MDS, SNE, t-SNE, etc. only care about pairwise distances between points. They try to position the points on a plane such that the ...
amoeba's user avatar
  • 106k
23 votes
Accepted

What is the connection between partial least squares, reduced rank regression, and principal component regression?

These are three different methods, and none of them can be seen as a special case of another. Formally, if $\mathbf X$ and $\mathbf Y$ are centered predictor ($n \times p$) and response ($n\times q$) ...
amoeba's user avatar
  • 106k
23 votes
Accepted

t-SNE versus MDS

PCA selects influential dimensions by eigenanalysis of the N data points themselves, while MDS selects influential dimensions by eigenanalysis of the $N^2$ data points of a pairwise distance matrix. ...
aminorex's user avatar
  • 346
20 votes
Accepted

Choosing the hyperparameters using T-SNE for classification

I routinely use $t$-SNE (alongside clustering techniques - more on this in the end) to recognise/assess the presence of clusters in my data. Unfortunately to my knowledge there is no standard way to ...
usεr11852's user avatar
  • 44.8k
20 votes

How can I reduce a very large sample size for statistical significance (sampling methods)?

Hypothesis testing with p-values is useful in some situations where you need to make a crisp decision from one experiment. You don't. Instead of worrying about p-values, compute confidence intervals ...
Harvey Motulsky's user avatar
19 votes
Accepted

Does "curse of dimensionality" really exist in real data?

This paper(1) discusses the blessing of non-uniformity as a counterpoint to the curse of dimensionality. The main idea is that data are not uniformly dispersed within the feature space, so one can ...
Sycorax's user avatar
  • 92.3k
19 votes

Why is dimensionality reduction always done before clustering?

Clustering generally depends on some sort of distance measure. Points near each other are in the same cluster; points far apart are in different clusters. But in high dimensional spaces, distance ...
G5W's user avatar
  • 2,640
18 votes
Accepted

Understanding this PCA plot of ice cream sales vs temperature

I know that PCA objective is to reduce dimensionality This is often what people assume, but in fact PCA is just a representation of your data onto an orthogonal basis. This basis still has the same ...
ilanman's user avatar
  • 4,787
18 votes

Should dimensionality reduction for visualization be considered a "closed" problem, solved by t-SNE?

Definitely not. I agree that t-SNE is an amazing algorithm that works extremely well and that was a real breakthrough at the time. However: it does have serious shortcomings; some of the ...
amoeba's user avatar
  • 106k
17 votes
Accepted

PCA vs. random projection

PCA maintains the best possible projection. Some reasons you would use random projections are: With very high dimensions, if speed is an issue, then consider that on a matrix of size $n \times k$, ...
ilanman's user avatar
  • 4,787
17 votes

Is PCA always recommended?

First of all, blindly throwing a model on some data cannot be possibly recommended (you may be able to relax that no-no if you have an infinite amount of independent cases at hand...). There is a ...
cbeleites unhappy with SX's user avatar
17 votes
Accepted

What does it mean when PCA does not produce a reduction in dimensionality?

Results suggest that your features are mutually orthogonal. Accounting for total variance means accounting for both variance and covariance. Orthogonality limits covariance. Standardization equates ...
Ed Rigdon's user avatar
  • 1,514
17 votes

Why does Harrell Argue for "Ignoring Y during data reduction"?

Does unsupervised learning not increase the risk of throwing out relevant variables? As Frank says, "Manipulations of X in unsupervised learning may result in a loss of information for ...
Demetri Pananos's user avatar
16 votes
Accepted

How to determine parameters for t-SNE for reducing dimensions?

I highly reccomend the article How to Use t-SNE Effectively. It has great animated plots of the tsne fitting process, and was the first source that actually gave me an intuitive understanding of what ...
Zach's user avatar
  • 24k
16 votes

A way to train a model on data with a very large number of features

Reshaping the data doesn't solve the problem because at the end of it, you have the same amount of data, plus an additional "index" column. If loading 1 row is expensive, then loading 1 row ...
Sycorax's user avatar
  • 92.3k
16 votes
Accepted

Why does Harrell Argue for "Ignoring Y during data reduction"?

All the answers are terrific. I'll just add a "big picture" note. We use data reduction not because it's perfect but when the alternative is a disaster. One of the worst things you can do ...
Frank Harrell's user avatar
15 votes

Functional principal component analysis (FPCA): what is it all about?

I worked for several years with Jim Ramsay on FDA, so I can perhaps add a few clarifications to @amoeba's answer. I think on a practical level, @amoeba is basically right. At least, that's the ...
Placidia's user avatar
  • 14.4k
15 votes

Explain "Curse of dimensionality" to a child

I have come across the following link that provides a very intuitive (and detailed) explanation of curse of dimensionality: http://www.visiondummy.com/2014/04/curse-dimensionality-affect-...
kostas's user avatar
  • 411

Only top scored, non community-wiki answers of a minimum length are eligible