I have a huge dataset (50,000 2000-dimensional sparse feature vectors). I want to cluster them in to k (unknown)clusters. As hierarchical clustering is very expensive in terms of time complexity (though it provides better result), I have designed my clustering framework as follows:
- do K-means clustering to partition the data into several bins (k is unknown so I make it reasonably large. eg. k=500)
- get centroids of all 500 partitions
- do hierarchical clustering on those 500 centroids(kind of merging based on some threshold value t)
- assign the datapoints to the nearest centroid(centroids emerged from hierarchical clustering)
I would like to know, whether my approach is efficient and if possible any other good solution to this problem.