I'm working on clustering for 6 month right now and there is question that bothers me lately, and that is why in every single resource about hierarchical clustering someone introduced two types of it (Agglomerative and Divisive) and then said "Hey, We only focus on Agglomerative"!
In my project I have to use hierarchical clustering and for now I'm using K-Means to divide the observations and in each new cluster I run K-means again till I reached the criteria.
There is a problem here; as long as number of observation are quit large (26000 for now) and each observation has 344 feature (output of PCA), I cannot follow agglomerative method and I prefer divisive method. For now I choose number of cluster as a constant (equal to 7) and clustering till I have 3 levels. Because of these constraints my model is not quit well. So I decided to use Elbow Method to determine number of clusters in each step.
Now after this long introduction, I came up with some problems:
- Using elbow method is too expensive. From here I'm choosing optimized
kin each set of data. But as elbow method wants inertia (intra cluster distance) for every single
k, I have to calculate this statistics for each set of data. This is a very time consuming task and choosing maximum number of
kis, let's say, risky.
- An important issue besides number of clusters is stopping criteria. For now I said "If number of observation in a node is under 100, Stop!". But you know this doesn't work very well. So, what is the best stopping criteria for divisive method?
After all of these, is there any methods that implemented divisive hierarchical clustering ?
If you need a clarification, just simply ask for it. Thank You.