# Tool form Hierarchical clustering

I'm trying to perform a hierarchical Clustering Analysis in a dataset of 40 attributes and +70,000 records, which is mostly composed by categorical variables. I've used Matlab and RapidMiner to execute the analysis but among their poor peformance and the lack of consistent outputs, I still can not get any results.

Is it any constraint to use this clustering method for such a dataset like the one I described?

I need to perform this analysis as part of a data mining research, to apply feature selection algorithms and measure performance gains afterwards.

• If you are clustering records, 70000 is too much for hierarchical method. First, 70000x70000 distance matrix is huge: will you computer or implementation cope? Another reason why hierarchical clustering is bad with thousands of records is that the risk/amount of suboptimality of the greedy algo can apper quite large on such datasets. Consider doing clustering by portions or using methods such as Two-step cluster which are designed for huge datasets. – ttnphns Oct 7 '14 at 7:05

I have used single-link clustering on 100k objects with ELKI. The SLINK implementation there is reasonably fast - it should complete in 10 minutes or so on this size of data. It doesn't keep a 70000x70000 matrix in memory (SLINK is $O(n^2)$ runtime, $O(2n)$ memory - much better than the naive algorithm which is $O(n^3)$ runtime and $O(n^2)$ memory - but also note that SLINK can only do single-linkage clustering). But categorial variable support isn't available in the download package - you would need to implement the parser, data type and the distance function yourself (which isn't hard, because there is a Tutorial that worked for me). Or you encode your categories as numbers, and use one of the included distance functions.