Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

248 questions
Filter by
Sorted by
Tagged with
71k views

Why is Euclidean distance not a good metric in high dimensions?

I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ...
30k views

Explain “Curse of dimensionality” to a child

I heard many times about curse of dimensionality, but somehow I'm still unable to grasp the idea, it's all foggy. Can anyone explain this in the most intuitive way, as you would explain it to a child,...
26k views

Best PCA algorithm for huge number of features (>10K)?

I previously asked this on StackOverflow, but it seems like it might be more appropriate here, given that it didn't get any answers on SO. It's kind of at the intersection between statistics and ...
57k views

How to estimate shrinkage parameter in Lasso or ridge regression with >50K variables?

I want to use Lasso or ridge regression for a model with more than 50,000 variables. I want do so using software package in R. How can I estimate the shrinkage parameter ($\lambda$)? Edits: Here is ...
2k views

Should dimensionality reduction for visualization be considered a “closed” problem, solved by t-SNE?

I've been reading a lot about $t$-sne algorithm for dimensionality reduction. I'm very impressed with the performance on "classic" datasets, like MNIST, where it achieves a clear separation of the ...
551 views

Why is LASSO not finding my perfect predictor pair at high dimensionality?

I'm running a small experiment with LASSO regression in R to test if it is able to find a perfect predictor pair. The pair is defined like this: f1 + f2 = outcome The outcome here is a predetermined ...
5k views

Should data be centered+scaled before applying t-SNE?

Some of my data's features have large values, while other features have much smaller values. Is it necessary to center+scale data before applying t-SNE to prevent bias towards the larger values? I ...
3k views

Does “curse of dimensionality” really exist in real data?

I understand what is "curse of dimensionality", and I have done some high dimensional optimization problems and know the challenge of the exponential possibilities. However, I doubt if the "curse of ...
570 views

High-dimensional regression: why is $\log p/n$ special?

I am trying to read up on the research in the area of high-dimensional regression; when $p$ is larger than $n$, that is, $p >> n$. It seems like the term $\log p/n$ appears often in terms of ...
7k views

PCA on high-dimensional text data before random forest classification?

Does it make sense to do PCA before carrying out a Random Forest Classification? I'm dealing with high dimensional text data, and I want to do feature reduction to help avoid the curse of ...
4k views

Does Dimensionality curse effect some models more than others?

The places I have been reading about dimensionality curse explain it in conjunction to kNN primarily, and linear models in general. I regularly see top rankers in Kaggle using thousands of features on ...
4k views

Is Multiple Linear Regression in 3 dimensions a plane of best fit or a line of best fit?

Our prof is not getting into the math or even geometric representation of multiple linear regression and this has me slightly confused. On the one hand it's still called multiple linear regression, ...
435 views