Skip to main content
35 votes
Accepted

Should data be centered+scaled before applying t-SNE?

Centering shouldn't matter since the algorithm only operates on distances between points, however rescaling is necessary if you want the different dimensions to be treated with equal importance, since ...
jon_simon's user avatar
  • 2,049
34 votes
Accepted

Curse of dimensionality- does cosine similarity work better and if so, why?

Contrary to various unproven claims, cosine cannot be significantly better. It is easy to see that Cosine is essentially the same as Euclidean on normalized data. The normalization takes away one ...
Has QUIT--Anony-Mousse's user avatar
32 votes
Accepted

Mathematical demonstration of the distance concentration in high dimensions

There is a simple mathematical thought experiment that sheds light on this phenomenon, although it might not seem immediately applicable. I will therefore describe this experiment briefly and follow ...
whuber's user avatar
  • 327k
26 votes
Accepted

How do I know my k-means clustering algorithm is suffering from the curse of dimensionality?

It helps to think about what The Curse of Dimensionality is. There are several very good threads on CV that are worth reading. Here is a place to start: Explain “Curse of dimensionality” to a child....
gung - Reinstate Monica's user avatar
26 votes

Why is Gaussian distribution on high dimensional space like a soap bubble

I can't answer about what the OP's famous post claims, but let us consider the simpler case of uniform distributions on the unit disc: $(X,Y)$ is uniformly distributed on the unit disc (that is, $f(X,...
Dilip Sarwate's user avatar
21 votes
Accepted

High-dimensional regression: why is $\log p/n$ special?

(Moved from comments to an answer as requested by @Greenparker) Part 1) The $\sqrt{\log p}$ term comes from (Gaussian) concentration of measure. In particular, if you have $p$ IID Gaussian random ...
mweylandt's user avatar
  • 831
19 votes
Accepted

Does "curse of dimensionality" really exist in real data?

This paper(1) discusses the blessing of non-uniformity as a counterpoint to the curse of dimensionality. The main idea is that data are not uniformly dispersed within the feature space, so one can ...
Sycorax's user avatar
  • 92.2k
18 votes

Should dimensionality reduction for visualization be considered a "closed" problem, solved by t-SNE?

Definitely not. I agree that t-SNE is an amazing algorithm that works extremely well and that was a real breakthrough at the time. However: it does have serious shortcomings; some of the ...
amoeba's user avatar
  • 106k
15 votes

Explain "Curse of dimensionality" to a child

I have come across the following link that provides a very intuitive (and detailed) explanation of curse of dimensionality: http://www.visiondummy.com/2014/04/curse-dimensionality-affect-...
kostas's user avatar
  • 411
15 votes

Functional principal component analysis (FPCA): what is it all about?

I worked for several years with Jim Ramsay on FDA, so I can perhaps add a few clarifications to @amoeba's answer. I think on a practical level, @amoeba is basically right. At least, that's the ...
Placidia's user avatar
  • 14.4k
15 votes

A way to train a model on data with a very large number of features

Reshaping the data doesn't solve the problem because at the end of it, you have the same amount of data, plus an additional "index" column. If loading 1 row is expensive, then loading 1 row ...
Sycorax's user avatar
  • 92.2k
14 votes

How do I know my k-means clustering algorithm is suffering from the curse of dimensionality?

My answer is not limit to K means, but check if we have curse of dimensionality for any distance based methods. K-means is based on a distance measure (for example, Euclidean distance) Before run the ...
Haitao Du's user avatar
  • 37.1k
14 votes
Accepted

Where are most points in a uniformly distributed high-dimensional ball?

As pointed out by @Xi'an, the OP's question is actually about a uniform distribution on the $n$-dimensional ball of radius $r$, the set of points at distance no more than $r$ from the center of the ...
Dilip Sarwate's user avatar
13 votes

PCA on high-dimensional text data before random forest classification?

I'd like to add my two cents to this since I thought the existing answers were incomplete. Performing PCA can be especially useful before training a random forest (or LightGBM, or any other decision ...
Ivan Batalov's user avatar
13 votes
Accepted

Is Multiple Linear Regression in 3 dimensions a plane of best fit or a line of best fit?

You're right, the solution surface is going to be a hyperplane in general. It's just that the word hyperplane is a mouthful, plane is shorter, and line is even shorter. As you continue on in math, ...
Matthew Drury's user avatar
13 votes

Mathematical demonstration of the distance concentration in high dimensions

Note that this depends on a) the distance measure (you are probably referring to the Euclidean distance) and b) the underlying measure/probability distribution, according to which you specify what "...
Christian Hennig's user avatar
13 votes

Dimension reduction using space filling curve to avoid "Curse of dimensionality"?

I think your intuition is right; moving from $\mathbb{R}^n$ to an affine parameter along a space-filling curve will discard information about what points are close to one another in the high-...
Nobody's user avatar
  • 2,045
12 votes
Accepted

Things that I am not sure about "LASSO" regression method

For the first question, recall that in centering we replace each value $y_i$ with $y_i - \bar y$, where $\bar y$ is the mean of the $y$ vector. Then $$ \sum_i (y_i - \bar y) = \sum_i y_i - n \bar y =...
Matthew Drury's user avatar
12 votes
Accepted

Can I use lasso when it is not a high dimensional setting?

There's nothing that suggests you need a number of predictors ($p$) as large as 200 or sample size ($n$) as large 500, let alone larger. (You might find it surprising to read some of the early papers ...
Glen_b's user avatar
  • 285k
12 votes
Accepted

What do high dimensional cauchy distributions look like?

The main challenge in this question lies in interpreting the sense of "accumulate around some manifold." The difficulty is that no such thing can happen, because as the vector length $d$ ...
whuber's user avatar
  • 327k
12 votes
Accepted

Understand the illustration of the curse of dimensionality?

Let's look at the first few dimensions. For $d=1$, if examples are laid out on a regular grid, this just means that they are at equal distances on a straight line, e.g., at the integers. We can ...
Stephan Kolassa's user avatar
11 votes
Accepted

How can I quickly detect cheating variables in large data?

This is sometimes referred to as "Data Leakage." There's a nice paper on this here: Leakage in Data Mining: Formulation, Detection, and Avoidance The above paper has plenty of amusing (and ...
Alex R.'s user avatar
  • 14k
11 votes
Accepted

Why do I get a non-zero intercept using the lasso even though I centered the response?

It is not in general the case that centering $y$ produces an intercept of zero. Centering $y$ means that the average value of the centered variable is zero, but the intercept is the predicted value ...
jkpate's user avatar
  • 1,449
10 votes
Accepted

How do children manage to pull their parents together in a PCA projection of a GWAS data set?

During the discussion with @ttnphns in the comments above, I realized that the same phenomenon can be observed with many fewer than 10 families. Three families (n=3 ...
amoeba's user avatar
  • 106k
10 votes

Why is Gaussian distribution on high dimensional space like a soap bubble

The post you link to concerns the use of the normal distribution in high-dimensional problems. So, suppose you are working in a space $\mathbb{R}^m$ where the dimension $m$ is large. Let $\...
Ben's user avatar
  • 127k
10 votes
Accepted

Why does the condition number of the covariance matrix explode as number of variables increases?

Explaining this in the comments was a little limiting, apologies: Assuming centered data matrix $X$, then your covariance matrix $M = X^T X$. This will have high condition number if the range of ...
proof_by_accident's user avatar
9 votes

Does "curse of dimensionality" really exist in real data?

Curse of dimensionality in machine learning is more often the problem of exploding empty space between the few data points that you have. Low manifold data can make it even worse. Here is an example ...
Gere's user avatar
  • 2,071
9 votes
Accepted

Different definitions of "curse of dimensionality"

It is not a mathematical object like a derivative that needs to be defined formally without any ambiguity. It is an umbrella term for those two issues encountered when using high dimensional data. The ...
David Ernst's user avatar
  • 3,189
9 votes
Accepted

PCA too slow when both n,p are large: Alternatives?

Question 1: Let's say you have observed a data matrix $X \in \mathbb R^{n \times p}$. From this you can compute the eigendecomposition $X^T X = Q \Lambda Q^T$. The question now is: if we get new data ...
jld's user avatar
  • 20.4k
9 votes
Accepted

Does SVM suffer from curse of high dimensionality? If no, Why?

SVM also suffers the problems coming from high dimensionality, but under typical settings to a lesser degree compared to (say) LDA. I can imagine SVM would only have to take dot products of support ...
Karolis Koncevičius's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible