3
$\begingroup$

I'm trying to understand the symptom of sparsity when increasing the number of features. I followed this article

In this article, the writer claims (refer to Figure 10 please) that

These samples are difficult to classify because their feature values greatly differs (e.g. samples in opposite corners of the unit square).

Then, obviously when increasing the dimension (Figure 11), the ratio between the sphere and the feature space is increased.

The author says that

In high dimensional spaces, most of the training data resides in the corners of the hypercube defining the feature space. As mentioned before, instances in the corners of the feature space are much more difficult to classify than instances around the centroid of the hypersphere

So mainly I don't understand two things:

  1. Why is it harder to classify "cornered samples"? Aren't they more easy to classify because some features are more significance?
  2. Why is it true that "In high dimensional spaces, most of the training data resides in the corners of the hypercube"?

Thanks

$\endgroup$

1 Answer 1

1
$\begingroup$

Regarding question 2, it looks like the author is assuming that the data are distributed uniformly throughout the training space. Then, since the training space has volume 1 by assumption, the fraction of data in any given subspace $X$ is approximately the volume of $X$. And since the volume of the unit hypersphere goes to 0 as the number of dimensions goes to infinity, the fraction of data in the unit hypersphere goes to 0.

$\endgroup$
1
  • $\begingroup$ Thank you. Do you happen to have some insights on q1? $\endgroup$
    – Covvar
    Commented Jul 30, 2017 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.