# Tag Info

Accepted

### How to reverse PCA and reconstruct original variables from several principal components?

PCA computes eigenvectors of the covariance matrix ("principal axes") and sorts them by their eigenvalues (amount of explained variance). The centered data can then be projected onto these principal ...
• 106k
Accepted

### What is "reduced-rank regression" all about?

1. What is reduced-rank regression (RRR)? Consider multivariate multiple linear regression, i.e. regression with $p$ independent variables and $q$ dependent variables. Let $\mathbf X$ and $\mathbf Y$ ...
• 106k

### Relationship between SVD and PCA. How to use SVD to perform PCA?

I wrote a Python & Numpy snippet that accompanies @amoeba's answer and I leave it here in case it is useful for someone. The comments are mostly taken from @amoeba's answer. ...
Accepted

### Why is t-SNE not used as a dimensionality reduction technique for clustering or classification?

The main reason that $t$-SNE is not used in classification models is that it does not learn a function from the original space to the new (lower) dimensional one. As such, when we would try to use our ...
• 44.8k
Accepted

### Is PCA always recommended?

Blindly using PCA is a recipe for disaster. (As an aside, automatically applying any method is not a good idea, because what works in one context is not guaranteed to work in another. We can formalize ...
• 92.3k
Accepted

### Should data be centered+scaled before applying t-SNE?

Centering shouldn't matter since the algorithm only operates on distances between points, however rescaling is necessary if you want the different dimensions to be treated with equal importance, since ...
• 2,049

### Relationship between SVD and PCA. How to use SVD to perform PCA?

Let me start with PCA. Suppose that you have $n$ data points comprised of $d$ numbers (or dimensions) each. If you center this data (subtract the mean data point $\mu$ from each data vector $x_i$) you ...
• 511
Accepted

### Intuitive explanation of how UMAP works, compared to t-SNE

You said that your understanding of t-SNE is based on https://www.youtube.com/watch?v=NEaUSP4YerM and you are looking for an explanation of UMAP on a similar level. I watched this video and it is ...
• 106k
Accepted

### Nystroem Method for Kernel Approximation

Let's derive the Nyström approximation in a way that should make the answers to your questions clearer. The key assumption in Nyström is that the kernel function is of rank $m$. (Really we assume that ...
• 25.1k
Accepted

### How can t-SNE or UMAP embed new (test) data, given that they are nonparametric?

Great question. I will answer it using t-SNE because I assume it is familiar to more people. I think UMAP is very promising and is a great contribution but to be honest I am getting a little bit ...
• 106k

### Intuitive explanation of how UMAP works, compared to t-SNE

The main difference between t-SNE and UMAP is the interpretation of the distance between objects or "clusters". I use the quotation marks since both algorithms are not meant for clustering - they are ...
• 1,701
Accepted

### Modelling with more variables than data points

It's certainly possible to fit good models when there are more variables than data points, but this must be done with care. When there are more variables than data points, the problem may not have a ...
• 32.8k
Accepted

### What is the meaning of the axes in t-SNE?

Individual axes in t-SNE have no meaning at all. Algorithms such as MDS, SNE, t-SNE, etc. only care about pairwise distances between points. They try to position the points on a plane such that the ...
• 106k
Accepted

### What is the connection between partial least squares, reduced rank regression, and principal component regression?

These are three different methods, and none of them can be seen as a special case of another. Formally, if $\mathbf X$ and $\mathbf Y$ are centered predictor ($n \times p$) and response ($n\times q$) ...
• 106k
Accepted

### t-SNE versus MDS

PCA selects influential dimensions by eigenanalysis of the N data points themselves, while MDS selects influential dimensions by eigenanalysis of the $N^2$ data points of a pairwise distance matrix. ...
• 346
Accepted

### Choosing the hyperparameters using T-SNE for classification

I routinely use $t$-SNE (alongside clustering techniques - more on this in the end) to recognise/assess the presence of clusters in my data. Unfortunately to my knowledge there is no standard way to ...
• 44.8k

### How can I reduce a very large sample size for statistical significance (sampling methods)?

Hypothesis testing with p-values is useful in some situations where you need to make a crisp decision from one experiment. You don't. Instead of worrying about p-values, compute confidence intervals ...
• 20.8k
Accepted

### Does "curse of dimensionality" really exist in real data?

This paper(1) discusses the blessing of non-uniformity as a counterpoint to the curse of dimensionality. The main idea is that data are not uniformly dispersed within the feature space, so one can ...
• 92.3k

### Why is dimensionality reduction always done before clustering?

Clustering generally depends on some sort of distance measure. Points near each other are in the same cluster; points far apart are in different clusters. But in high dimensional spaces, distance ...
• 2,640
Accepted

### Understanding this PCA plot of ice cream sales vs temperature

I know that PCA objective is to reduce dimensionality This is often what people assume, but in fact PCA is just a representation of your data onto an orthogonal basis. This basis still has the same ...
• 4,787

### Should dimensionality reduction for visualization be considered a "closed" problem, solved by t-SNE?

Definitely not. I agree that t-SNE is an amazing algorithm that works extremely well and that was a real breakthrough at the time. However: it does have serious shortcomings; some of the ...
• 106k
Accepted

### PCA vs. random projection

PCA maintains the best possible projection. Some reasons you would use random projections are: With very high dimensions, if speed is an issue, then consider that on a matrix of size $n \times k$, ...
• 4,787

### Is PCA always recommended?

First of all, blindly throwing a model on some data cannot be possibly recommended (you may be able to relax that no-no if you have an infinite amount of independent cases at hand...). There is a ...
Accepted

### What does it mean when PCA does not produce a reduction in dimensionality?

Results suggest that your features are mutually orthogonal. Accounting for total variance means accounting for both variance and covariance. Orthogonality limits covariance. Standardization equates ...
• 1,514

### Why does Harrell Argue for "Ignoring Y during data reduction"?

Does unsupervised learning not increase the risk of throwing out relevant variables? As Frank says, "Manipulations of X in unsupervised learning may result in a loss of information for ...
• 37.6k
Accepted

### How to determine parameters for t-SNE for reducing dimensions?

I highly reccomend the article How to Use t-SNE Effectively. It has great animated plots of the tsne fitting process, and was the first source that actually gave me an intuitive understanding of what ...
• 24k

### A way to train a model on data with a very large number of features

Reshaping the data doesn't solve the problem because at the end of it, you have the same amount of data, plus an additional "index" column. If loading 1 row is expensive, then loading 1 row ...
• 92.3k
Accepted

### Why does Harrell Argue for "Ignoring Y during data reduction"?

All the answers are terrific. I'll just add a "big picture" note. We use data reduction not because it's perfect but when the alternative is a disaster. One of the worst things you can do ...
• 95.1k