Currently I am studying effect of high dimensions of data on clustering , for experiment purpose I want to use kdd dataset from UCI which contains 42 features. Is kdd a high dimensional data or what is the threshold of number of dimensions beyond that we can say data is high dimensional ?

  • 1
    $\begingroup$ High dimensional, as indicated by the question, is subject to opinion. In my experience, in machine learning we consider anything less than thousands of dimensions low dimensional, but opinions will vary. $\endgroup$ – Marc Claesen Apr 1 '16 at 6:05
  • $\begingroup$ Yes , it may vary as per opinion. I am searching "Is there any strong reference (research paper ) which discuses this issue in detail?" $\endgroup$ – swapnil7 Apr 1 '16 at 6:15
  • $\begingroup$ You need to look at stuff in your community and see how they define "high dimensionality". Then find the expected value of "high dimensionality" as you weight by the importance of the person who utters it. I dunno this dude, but he tests against high dimensionality by trying increasing dimensions from low (hundreds of dimensions) up to 13k dimensions (which they consider high dimensional) togaware.com/papers/ijdwm2012.pdf $\endgroup$ – caveman Apr 1 '16 at 8:07
  • $\begingroup$ The question "what is high-dimensional" is discussed tangentially in this thread, but I don't think it's a duplicate question. Just also of interest: stats.stackexchange.com/questions/99171/… $\endgroup$ – Sycorax Apr 1 '16 at 19:28

A way to see high dimension, is when there are more regressors/predictors than observations. If $p$ denotes the number of regressors and $n$ the number of observations, high dimension is when $p > n$ and even $p >> n$. If I remember well, penalized regressions (ridge, lasso) have been introduced partly in order to tackle this issue (classical OLS in this setting to do not give a unique solution).

Edit : As asked, some details about what I said. And I apologize about the fact that what I'm talking about is more relevant in a supervised framework. This definition (which can be thought as subjective of course) is the consequence of the following. If you consider a classical linear regression : $Y = X\beta + \epsilon$ then you have the OLS estimator : $\hat\beta = (X'X)^{-1}X'Y$ which is valid only if $(X'X)^{-1}$ exists. Or if $dim(X)=(n,p)$ with $n < p$ then $X'X$ is not full rank, then cannot be inverted and then no more $\hat\beta$ as previously. So switching from $n>p$ to $p > n$ is not trivial. Or multiple linear regression is widely used (especially in econometrics for instance, but also in epidemiology when studying genes...) hence I think it is a convenient way to define "high dimension" (but that's true : it's subjective) because you need to do something different from what you usually do.

For clustering, maybe it can be seen differently, with k-nearest neighbors curse of dimensionality is reached a long time before $p > n$...

Some references :

| cite | improve this answer | |
  • $\begingroup$ Can you explain why this definition is more helpful than others? If it's subjective, then can you cite things that shows the kind of community that follow this definition? $\endgroup$ – caveman Apr 1 '16 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.