6
$\begingroup$

Is there a simple statistical test that I can use to determine whether my data is spherically separable? I am planning to use Kmeans++ to divide 48 dimensional vectors into clusters but I just read that this depends on the assumption that my data is spherically separable…

$\endgroup$
1
  • 2
    $\begingroup$ The "test" is to try it and see if it works. $\endgroup$
    – Neil G
    Aug 16 '15 at 11:15
1
$\begingroup$

I think the best and easiest thing you can do when you have data is to just implement your model (k-means), train your model, and then validate your model on unseen data. The validation error tells you how good your model is. You can safely compare any number of models this way.

Visualization might work for small models, but it's really hard to project the 48-dimensional vectors you have to 2 dimensions and expect to see class separations. Essentially, your k-means is doing a projection already.

Other answers are pointing out that k-means makes assumptions. All models make assumptions. If they make the wrong assumptions, then that will be revealed when you validate.

$\endgroup$
2
$\begingroup$

The two main approaches are:

  1. Visualize (yes, there are methods)
  2. try clustering and evaluate carefully on your data

Do not rely on any automatic method or statistic.

$\endgroup$
4
  • $\begingroup$ Ok. But, I read here(r-bloggers.com/k-means-clustering-is-not-a-free-lunch) that it also assumes that all variables have the same variance. So I should be able to use Bartlett's test for this. But, I don't know of any tests that determine whether the variance of the distribution of each variable is spherical. Would you have any idea how I may do this? $\endgroup$ Aug 16 '15 at 13:33
  • $\begingroup$ Again, do not rely on tests. Visualize, and evaluate. The tests work pn toy data, and return garbage on real data. $\endgroup$ Aug 16 '15 at 14:13
  • $\begingroup$ When you say visualize, what methods would you recommend? I've already tried multidimensional scaling and using bar charts to look at the average variance explained by each cluster center. $\endgroup$ Aug 17 '15 at 8:57
  • 1
    $\begingroup$ MDS is worth a try, but also try methods that use class information, such as Fisher's linear discriminant. I have heard that t-SNE is also a very good technique, but I have never tried it. $\endgroup$ Aug 17 '15 at 13:49
0
$\begingroup$

Using this blog post as a reference it appears that it's possible to do better than 'try clustering' and 'visualize':

1) all variables should have the same variance so I can use Bartlett's test on all variables.

2) the prior probability for all k clusters are the same (i.e. each cluster has roughly equal number of observations) and this is something I can check as well.

3) k-means assume the variance of the distribution of each variable is spherical

Now, I'm not sure how to test point 3 which is my question. But, at least these three conditions must hold. So I am not limited to checking whether the variance of the distribution of each variable is spherical.

$\endgroup$
11
  • $\begingroup$ I don't agree with this answer: Just because the prior is flat, doesn't mean the k-means won't work well on data that heavily favors one category or another. Data can always overwhelm priors. I'll add an answer. $\endgroup$
    – Neil G
    Aug 16 '15 at 14:25
  • $\begingroup$ Does this mean that this particular assumption is of little importance? $\endgroup$ Aug 16 '15 at 14:37
  • $\begingroup$ That assumption is the same as the one wherein a multinomial logistic regression model's "prior" over the categories is uniform. It doesn't mean that multinomial logistic regression only works well when data is evenly distributed between the categories. $\endgroup$
    – Neil G
    Aug 16 '15 at 14:39
  • 2
    $\begingroup$ Consider $n$ categories contained in perfect spheres. Now add a small number of very distant outliers. k-means will still perform extremely well (few misclassifications), but you can make the class variances as large as you like. $\endgroup$
    – Neil G
    Aug 16 '15 at 15:05
  • 1
    $\begingroup$ Thank you for presenting this counterexample and I appreciated your patience. I'll look more closely at my data in this case. $\endgroup$ Aug 16 '15 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.