Questions tagged [high-dimensional]

Pertains to a large number of features or dimensions (variables) for data. (For a large number of data points, use the tag [large-data]; if the issue is a larger number of variables than data, use the [underdetermined] tag.)

Filter by
Sorted by
Tagged with
236
votes
8answers
71k views

Why is Euclidean distance not a good metric in high dimensions?

I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ...
54
votes
7answers
26k views

Best PCA algorithm for huge number of features (>10K)?

I previously asked this on StackOverflow, but it seems like it might be more appropriate here, given that it didn't get any answers on SO. It's kind of at the intersection between statistics and ...
9
votes
2answers
6k views

How do I know my k-means clustering algorithm is suffering from the curse of dimensionality?

I believe that the title of this question says it all.
90
votes
11answers
30k views

Explain “Curse of dimensionality” to a child

I heard many times about curse of dimensionality, but somehow I'm still unable to grasp the idea, it's all foggy. Can anyone explain this in the most intuitive way, as you would explain it to a child,...
36
votes
3answers
57k views

How to estimate shrinkage parameter in Lasso or ridge regression with >50K variables?

I want to use Lasso or ridge regression for a model with more than 50,000 variables. I want do so using software package in R. How can I estimate the shrinkage parameter ($\lambda$)? Edits: Here is ...
16
votes
4answers
3k views

Does “curse of dimensionality” really exist in real data?

I understand what is "curse of dimensionality", and I have done some high dimensional optimization problems and know the challenge of the exponential possibilities. However, I doubt if the "curse of ...
13
votes
3answers
7k views

PCA on high-dimensional text data before random forest classification?

Does it make sense to do PCA before carrying out a Random Forest Classification? I'm dealing with high dimensional text data, and I want to do feature reduction to help avoid the curse of ...
0
votes
1answer
996 views

How is the lasso orthogonal design case solution derived?

In orthogonal design of lasso, we get $\hat{\beta}_j^{\text{lasso}} = 0 \text{ if abs}(\hat{\beta}_j) \le \lambda /2$. WHY? I've seen the answer and derived it myself, but don't know why. We begin ...
12
votes
1answer
4k views

Does Dimensionality curse effect some models more than others?

The places I have been reading about dimensionality curse explain it in conjunction to kNN primarily, and linear models in general. I regularly see top rankers in Kaggle using thousands of features on ...
7
votes
2answers
357 views

Generating a high-dimensional dataset where nearest neighbor becomes meaningless

In the paper "When Is 'Nearest Neighbor' Meaningful?" we read that, We show that under certain broad conditions (in terms of data and query distributions, or workload), as dimensionality increases,...
6
votes
2answers
734 views

Explanation for this event on a high-dimensional dataset

Suppose we sample a set $S$ of $n$ points from a $d$-dimensional spherical (unit variance) Gaussian with $d \approx 100$. It is known that any point of the sample would be roughly at $\sqrt{d}$ ...
3
votes
4answers
502 views

PCA, dimensionality, and k-means results: reaction to duplicating of variables

There are many excellent conversations on CV about the curse of dimensionality when applied to methods like k-means. The answer in the same post and other research (e.g., the paper titled "When Is ‘...
9
votes
3answers
3k views

Curse of dimensionality: kNN classifier

I am reading Kevin Murphy's book: Machine Learning-A probabilistic Perspective. In the first chapter the author is explaining the curse of dimensionality and there is a part which i do not understand. ...
9
votes
2answers
285 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
3
votes
0answers
403 views

Inference for quasibinomial GLM with LASSO penalty using selectiveInference package

I would like to carry out inference on a binomial LASSO model, but take into account the fact that my data are overdispersed and use the quasibinomial family instead. R package ...
9
votes
7answers
1k views

Find close pairs in very high dimensional space with sparse vectors

I have $N$ (~a million) feature vectors. There are $M$ (~a million) binary features, but in each vector only $K$ (~a thousand) of them would be $1$, the rest are $0$. I'm looking for the pairs of ...
9
votes
3answers
669 views

PCA too slow when both n,p are large: Alternatives?

Problem Setup I have data points (images) of high dimension (4096), which I'm trying to visualize in 2D. To this end, I'm using t-sne in a manner similar to the following example code by Karpathy. ...
4
votes
2answers
137 views

Why $p > n$ implies multicollinearity?

Why $p > n$ implies multicollinearity ? $p$ is number of variables, and $n$ is number of samples. I know it has something to do with linear algebra concepts, but I am not sure how do linear algebra ...
3
votes
0answers
522 views

Clustering in high dimensions: distance metrics, binary vs continuous, statistical tests for number of clusters / noise points [closed]

I’ve got several thousand observations in approximately 300-dimensional space, in a relatively sparse matrix (typically 30 non-zero dimensions per observation). I'm using a clustering algorithm (so ...
2
votes
2answers
214 views

How exactly does curse of dimensionality curse?

In what way does the curse of dimensionality affect the predictions? I know that as the number of predictors increases the observations that are geometrically near decrease, so we have to spread out ...
0
votes
0answers
38 views

Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
4
votes
1answer
568 views

Things that I am not sure about “LASSO” regression method

I have read the chapters that are related to "LASSO" regression in: The elements of statistical learning (Tibshirani et al.) Statistical Learning with Sparsity: The Lasso and Generalizations. (...
3
votes
1answer
2k views

Dispersion of points on 2D or 3D

Suppose that we have a set of points on a line. The amount of dispersion can be measured by standard deviation in this case. My question is, is there something similar for higher dimensions? For ...
2
votes
1answer
121 views

estimate precision matrix with given spatial sparsity pattern

I have a set of $n$ measurements of $p$ variables $\xi_i$. I am interested in the inverse covariance or precision matrix $P$ of the variables, but because $p \gg n$ and because of limited storage ($p$ ...
1
vote
2answers
101 views

Exact matching + multiple regression on high-dimensional treatment-control study?

I'm working on a project with healthcare data where episodes of care in the treatment and control groups must be matched to estimate average treatment effect (ATE). I have several hundred covariates ...