Currently I am studying effect of high dimensions of data on clustering , for experiment purpose I want to use kdd dataset from UCI which contains 42 features. Is kdd a high dimensional data or what is the threshold of number of dimensions beyond that we can say data is high dimensional ?
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ High dimensional, as indicated by the question, is subject to opinion. In my experience, in machine learning we consider anything less than thousands of dimensions low dimensional, but opinions will vary. $\endgroup$– Marc ClaesenCommented Apr 1, 2016 at 6:05
-
$\begingroup$ Yes , it may vary as per opinion. I am searching "Is there any strong reference (research paper ) which discuses this issue in detail?" $\endgroup$– swapnil7Commented Apr 1, 2016 at 6:15
-
$\begingroup$ You need to look at stuff in your community and see how they define "high dimensionality". Then find the expected value of "high dimensionality" as you weight by the importance of the person who utters it. I dunno this dude, but he tests against high dimensionality by trying increasing dimensions from low (hundreds of dimensions) up to 13k dimensions (which they consider high dimensional) togaware.com/papers/ijdwm2012.pdf $\endgroup$– cavemanCommented Apr 1, 2016 at 8:07
-
$\begingroup$ The question "what is high-dimensional" is discussed tangentially in this thread, but I don't think it's a duplicate question. Just also of interest: stats.stackexchange.com/questions/99171/… $\endgroup$– Sycorax ♦Commented Apr 1, 2016 at 19:28
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
A way to see high dimension, is when there are more regressors/predictors than observations. If $p$ denotes the number of regressors and $n$ the number of observations, high dimension is when $p > n$ and even $p >> n$. If I remember well, penalized regressions (ridge, lasso) have been introduced partly in order to tackle this issue (classical OLS in this setting to do not give a unique solution).
Edit : As asked, some details about what I said. And I apologize about the fact that what I'm talking about is more relevant in a supervised framework. This definition (which can be thought as subjective of course) is the consequence of the following. If you consider a classical linear regression : $Y = X\beta + \epsilon$ then you have the OLS estimator : $\hat\beta = (X'X)^{-1}X'Y$ which is valid only if $(X'X)^{-1}$ exists. Or if $dim(X)=(n,p)$ with $n < p$ then $X'X$ is not full rank, then cannot be inverted and then no more $\hat\beta$ as previously. So switching from $n>p$ to $p > n$ is not trivial. Or multiple linear regression is widely used (especially in econometrics for instance, but also in epidemiology when studying genes...) hence I think it is a convenient way to define "high dimension" (but that's true : it's subjective) because you need to do something different from what you usually do.
For clustering, maybe it can be seen differently, with k-nearest neighbors curse of dimensionality is reached a long time before $p > n$...
Some references :
http://statweb.stanford.edu/~tibs/ElemStatLearn/ The Elements of Statistical Learning where you can find such a definition and a lot of other things
for an econometrics perspective, for instance this presentation : http://www.nber.org/econometrics_minicourse_2013/Chernozhukov.pdf or this paper http://arxiv.org/pdf/1105.2454v4.pdf High dimension problems is a very dynamic field of econometrics at the moment.
-
$\begingroup$ Can you explain why this definition is more helpful than others? If it's subjective, then can you cite things that shows the kind of community that follow this definition? $\endgroup$– cavemanCommented Apr 1, 2016 at 15:24