1
$\begingroup$

I am using agglomerative hierarchical clustering to cluster 200 variables based on their Manhattan distances. When calculating the average distance within/average distance between ratio, I was expecting declining values as the number of clusters is increasing, for the reason that (as far as I now) the algorithm tries to optimise these values (well, ok, maybe not exactly this specific ratio but..).

My question is: is it possible that this ratio is not declining while k is increasing (where k is the number of clusters)? Increasing the number of clusters does not always decreasing this ratio? (by reducing the distance within or/and increasing the distance between)?

ps. the method for the distance calculation is the "complete" method from hclust in R.

$\endgroup$
  • $\begingroup$ average distance within/average distance between ratio This is entirely not clear. $\endgroup$ – ttnphns Oct 30 '15 at 9:11
  • $\begingroup$ It's the average of the distances within each cluster to the average of the distances between two clusters (as defined by complete method from hclust in R). It is a good way to check your clustering results. $\endgroup$ – Kwnwps Oct 30 '15 at 16:46
  • $\begingroup$ You might be mixing up things... Complete linkage method has nothing to do with averages or ratios. $\endgroup$ – ttnphns Oct 30 '15 at 18:01
  • $\begingroup$ Please read it again. An average can be an average of distances. Also a ratio can be a ratio of average values. $\endgroup$ – Kwnwps Oct 30 '15 at 23:30
1
$\begingroup$

"Likely" by what assumptions on your data? I don't see how you could arrive at a meaningful probability here, I would avoid using probabilistic terminology here.

Also note that it makes a huge difference whether you are optimizing the shortest-link, average-link, complete-link or that ratio that you gave. Optimizing for one obviously does not guarantee monotonicity for another. In these terms, your question is not well specified, because you do not say what Linkage you use.

$\endgroup$
  • $\begingroup$ "likely" is not used here with any probabilistic meaning. I edited the question so there is no room for that misunderstanding. I also edited to provide information about the link, although I am not entirely sure that has an effect on this question. $\endgroup$ – Kwnwps Oct 30 '15 at 16:44
0
$\begingroup$

Behavior you are describing (the not declining average distance) is completely normal for hierarchical clustering, where the output quality depends on the linkage method used and, mainly, the preprocessing method applied to the data.

You are not making any reference to the number of instances you have in the data, but using 200 variables could create redundancies which should be addressed before using the clustering method, a feature selection procedure could help you in getting better outputs.

$\endgroup$
  • $\begingroup$ Thanks. I have almost 1000 for all variables (same amount for all). Can you propose a feature selection procedure? $\endgroup$ – Kwnwps Nov 3 '15 at 2:16
  • $\begingroup$ Using numerical data, I've got the best outputs using Spectral Feature Selection (SPEC), it's implemented in Matlab $\endgroup$ – formacero10 Nov 3 '15 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.