1
$\begingroup$

I am using agglomerative hierarchical clustering to cluster 200 variables based on their Manhattan distances. When calculating the average distance within/average distance between ratio, I was expecting declining values as the number of clusters is increasing, for the reason that (as far as I now) the algorithm tries to optimise these values (well, ok, maybe not exactly this specific ratio but..).

My question is: is it possible that this ratio is not declining while k is increasing (where k is the number of clusters)? Increasing the number of clusters does not always decreasing this ratio? (by reducing the distance within or/and increasing the distance between)?

ps. the method for the distance calculation is the "complete" method from hclust in R.

$\endgroup$
4
  • $\begingroup$ average distance within/average distance between ratio This is entirely not clear. $\endgroup$
    – ttnphns
    Oct 30, 2015 at 9:11
  • $\begingroup$ It's the average of the distances within each cluster to the average of the distances between two clusters (as defined by complete method from hclust in R). It is a good way to check your clustering results. $\endgroup$
    – Kwnwps
    Oct 30, 2015 at 16:46
  • $\begingroup$ You might be mixing up things... Complete linkage method has nothing to do with averages or ratios. $\endgroup$
    – ttnphns
    Oct 30, 2015 at 18:01
  • $\begingroup$ Please read it again. An average can be an average of distances. Also a ratio can be a ratio of average values. $\endgroup$
    – Kwnwps
    Oct 30, 2015 at 23:30

2 Answers 2

1
$\begingroup$

"Likely" by what assumptions on your data? I don't see how you could arrive at a meaningful probability here, I would avoid using probabilistic terminology here.

Also note that it makes a huge difference whether you are optimizing the shortest-link, average-link, complete-link or that ratio that you gave. Optimizing for one obviously does not guarantee monotonicity for another. In these terms, your question is not well specified, because you do not say what Linkage you use.

$\endgroup$
1
  • $\begingroup$ "likely" is not used here with any probabilistic meaning. I edited the question so there is no room for that misunderstanding. I also edited to provide information about the link, although I am not entirely sure that has an effect on this question. $\endgroup$
    – Kwnwps
    Oct 30, 2015 at 16:44
0
$\begingroup$

Behavior you are describing (the not declining average distance) is completely normal for hierarchical clustering, where the output quality depends on the linkage method used and, mainly, the preprocessing method applied to the data.

You are not making any reference to the number of instances you have in the data, but using 200 variables could create redundancies which should be addressed before using the clustering method, a feature selection procedure could help you in getting better outputs.

$\endgroup$
2
  • $\begingroup$ Thanks. I have almost 1000 for all variables (same amount for all). Can you propose a feature selection procedure? $\endgroup$
    – Kwnwps
    Nov 3, 2015 at 2:16
  • $\begingroup$ Using numerical data, I've got the best outputs using Spectral Feature Selection (SPEC), it's implemented in Matlab $\endgroup$ Nov 3, 2015 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.