Questions tagged [hierarchical-clustering]

Hierarchical cluster analysis is a method of cluster analysis which builds, by steps, a hierarchy of clusters, a dendrogram. Most popular is agglomerative hierarchical clustering (HAC) which starts from individual objects and collects them into bigger and bigger clusters.

Filter by
Sorted by
Tagged with
56 votes
3 answers
37k views

How to select a clustering method? How to validate a cluster solution (to warrant the method choice)?

One of the biggest issue with cluster analysis is that we may happen to have to derive different conclusion when base on different clustering methods used (including different linkage methods in ...
user avatar
  • 869
51 votes
2 answers
89k views

Choosing the right linkage method for hierarchical clustering

I am performing hierarchical clustering on data I've gathered and processed from the reddit data dump on Google BigQuery. My process is the following: Get the latest 1000 posts in /r/politics ...
user avatar
  • 611
34 votes
3 answers
114k views

How to interpret the dendrogram of a hierarchical cluster analysis

Consider the R example below: plot( hclust(dist(USArrests), "ave") ) What exactly does the y-axis "Height" mean? Looking at North Carolina and California (...
user avatar
  • 3,286
27 votes
1 answer
36k views

Using correlation as distance metric (for hierarchical clustering)

I would like to hierarchically cluster my data, but rather than using Euclidean distance, I'd like to use correlation. Also, since the correlation coefficient ranges from -1 to 1, with both -1 and 1 ...
user avatar
  • 373
21 votes
4 answers
24k views

How to understand the drawbacks of Hierarchical Clustering?

Can someone explain the pros and cons of Hierarchical Clustering? Does Hierarchical Clustering have the same drawbacks as K means? What are the advantages of Hierarchical Clustering over K means? ...
user avatar
21 votes
2 answers
4k views

Clustering -- Intuition behind Kleinberg's Impossibility Theorem

I've been thinking about writing a blog post on this interesting analysis by Kleinberg (2002) that explores the difficulty of clustering. Kleinberg outlines three seemingly intuitive desiderata for a ...
user avatar
  • 2,416
12 votes
1 answer
10k views

Hierarchical clustering with categorical variables

Can categorical variables be used in hierarchical clustering? I have heard only continuous variables are used, but have seen people discussing categorical variables may / may not be used as well. ...
user avatar
11 votes
4 answers
28k views

Choosing the number of clusters in hierarchical agglomerative clustering

I have a set of points that I want to cluster into groups according to a number of features computed. I have distance matrix containing the distances between all different pairs of points. I have ...
user avatar
11 votes
1 answer
36k views

How to interpret dendrogram height for clustering by correlation

Given the following data frame: ...
user avatar
10 votes
2 answers
6k views

Does k-means have any advantages over HDBSCAN expect for runtime?

I have recently learned about HDBSCAN (a fairly new method for clustering, not yet available in scikit-learn) and am really surprised at how good it is. The following picture illustrates that the ...
user avatar
  • 223
9 votes
2 answers
6k views

Do I need to remove duplicate objects for cluster analysis of objects?

I am doing a cluster analyis and I was wondering whether it is possible to remove duplicates from the data set - in order to increase performance. I work on tables where objects are in rows and ...
user avatar
  • 397
9 votes
2 answers
15k views

What is the interpretation of eps parameter in DBSCAN clustering?

I want to cluster lat-long data such that all clusters formed will have radius<=1000 meters Questions What is the actual meaning of eps parameter? Please given an example. Will setting eps=1000 ...
user avatar
9 votes
2 answers
2k views

Does a distance have to be a "metric" for an hierarchical clustering to be valid on it?

Let us say that we define a distance, which is not a metric, between N items. Based on this distance we then use an Agglomerative hierarchical clustering. Can we use each of the known algorithm (...
user avatar
  • 20.3k
8 votes
1 answer
30k views

Hierarchical clustering of correlation matrix

I have a correlation matrix of 8,854 * 8,854 size. These are Pearson correlation coefficient values in the matrix. I want to perform Hierarchical clustering and create good resolution images like I ...
user avatar
  • 81
8 votes
2 answers
2k views

Can sub-optimality of various hierarchical clustering methods be assessed or ranked?

Classic agglomerative hierarchical clustering methods are based on a greedy algorithm. This means that they (many of them) are prone to give sub-optimal solutions instead of the global optimum result, ...
user avatar
  • 52.9k
8 votes
2 answers
9k views

Hierarchical clustering with agnes - how to cut the tree?

I am working on a data.frame with both categorical and metric variables ...
user avatar
  • 397
8 votes
1 answer
1k views

Best practices in the selection of distance metric and clustering methods for gene expression data

I have been reading about this on various channels including here and Stack Exchange, but I'm still not sure how to choose the best approach for clustering gene expression data. As a Ph.D. molecular ...
user avatar
  • 631
8 votes
2 answers
864 views

Choosing the number of clusters - clustering validation criterions vs domain theoretical considerations

I often face the issue of having to choose a k number of clusters. The partition I end up choosing is more often based on visual and theoretical concerns rather than quality criteria. I have two ...
user avatar
  • 763
8 votes
1 answer
4k views

Pooling levels of categorical variables for regression trees

I have a data set I would like to do a regression analysis for. There are many features of both categorical and continuous types. One of the categorical features has many (>75) levels so this is an ...
user avatar
  • 521
7 votes
1 answer
2k views

Why are the cluster analysis results using raw data the same as the ones using PCA scores?

I have read around a lot and tried different ways to carry out my cluster analysis. In the first case, I have carried out a hierarchical cluster analysis on my raw data (200 watersheds and 16 ...
user avatar
7 votes
5 answers
7k views

K-Means Cluster has over 50% of the points in one cluster. How to optimize it?

I am running a clustering algorithm in Spark and I have to choose between K-Means and Bisecting-Kmeans. However the only thing that differes between the two is the runtime because the performance is ...
user avatar
  • 173
7 votes
2 answers
5k views

Is there an advantage to squaring dissimilarities when using Ward clustering?

Is there a reason to prefer squaring or not squaring the dissimilarities when clustering with Ward's method? The question is motivated by the following statement in the documentation for R's ...
user avatar
  • 899
6 votes
2 answers
498 views

Conditional intraclass correlation (ICC) from a linear mixed model as evidence for test-retest reliability?

In my experiment with two conditions (between-subjects design), participants completed a single-item scale three times: (1) before the experimental manipulation, (2) after the experimental ...
user avatar
  • 67
6 votes
1 answer
3k views

Determining similar users from hierarchical clustering

I use hierarchical clustering to cluster users which are similar to each other based on a Jaccard coefficient. I have now coded a solution to extract similar users based on hierarchical clustering: ...
user avatar
  • 607
6 votes
1 answer
235 views

Increasing multicollinearity in multilevel/hierarchical modeling?

I have a linear model with response variable $\textbf{y}$ and explanatory variable matrix $\textbf{X}$ for which coefficients $\textbf{b}$ are physically meaningful and worth estimating: \begin{...
user avatar
  • 789
5 votes
2 answers
852 views

Motivation for Ward's definition of error sum of squares (ESS)

Ward (1963) provides a commonly used criterion for hierarchical clustering. It's based on the following definition (p. 237): Given a set of ratings for 10 individuals, $\{2, 6, 5, 6, 2, 2, 2, 0, 0, ...
user avatar
  • 19.2k
5 votes
1 answer
8k views

Validate dendrogram in cluster analysis: What is the meaning of cophenetic correlation coefficient?

I want to calculate the cophenetic correlation coefficient. Reading previous posts Comparison of cophenetic correlation coefficients on different data sets On cophenetic correlation for dendrogram ...
user avatar
  • 332
5 votes
2 answers
5k views

Testing whether two datasets cluster similarly

Most questions about cluster analysis seem to come from people who have a single dataset and want to compare/quantify the similarity of different clustering approaches. This question is not that. ...
user avatar
  • 51
5 votes
2 answers
2k views

Validate Cluster Analysis by doing it on two subsamples

I am working on validating a cluster analysis. I have read somewhere the approach to cross-validate the cluster analysis. The link of the article is http://jonathantemplin.com/files/clustering/...
user avatar
  • 589
5 votes
2 answers
1k views

How to reduce the dimensionality of a similarity matrix (of categorical co-occurence counts)?

Our example person Azra has assigned (open-ended categories of her own choosing) to a fixed set of 35 items, recorded as logical values (...
user avatar
  • 722
5 votes
1 answer
1k views

Purpose of dendrogram and hierarchical clustering

This is likely a very naive question. I've lately been reading about hierarchical clustering algorithms, and various discussions about how to interpret dendrograms or find optimal heights for cutting ...
user avatar
  • 51
5 votes
1 answer
82 views

Definition and Taxonomy of Seasonal Time Series

I want to categorize a large number of time series into non-seasonal and seasonal divide the seasonal ones into a small number of subgroups by type of seasonality Are there any formal definitions/...
user avatar
  • 31.5k
5 votes
1 answer
11k views

How to determine which variables to be used for cluster analysis

I have about 10 variables (features) and want to do cluster analysis of cases (data points). I have a number of ideas about which variables to be included for cluster analysis: Plot the variables ...
user avatar
  • 439
5 votes
1 answer
3k views

Best clustering algorithm for real estate data

I want to cluster real estate data to determine average price patterns in city and rural regions. My data set contains size, number of dorms, bathrooms and coordinates of the properties. Which would ...
user avatar
  • 185
5 votes
0 answers
1k views

Adjusting Standard Error for Imputed/Generated Regressors

This is my first question, so I hope this is a valid question. I am surprised that I have seen only few questions (and no answer helping me out) referring to the adjustment of variance estimators in ...
user avatar
5 votes
0 answers
383 views

How to impose spatial continuity constraint onto clustering?

Statistics version: I have a few measurements of a function that takes three inputs and produces a few 2D fields of outputs: $f(a,b,c;x,y)$, with $f$ being a vector of several quantities. I would like ...
user avatar
4 votes
2 answers
6k views

How to determine which method is the most valid, reasonable clustering results?

Method 1: Cluster by K-means with initial centroid {27, 67.5} Method 2: Cluster by K-means with initial centroid {22.5, 60} Method 3: Agglomerative Clustering How can I know which method gives a ...
user avatar
4 votes
2 answers
673 views

Which unsupervised classification method to use next if hierarchical clustering gave bad results?

Purposes I need to perform a classification of weather stations taking into account the characteristics of intra-annual variability of some two climate indicators. There are 613 sites with monthly ...
user avatar
4 votes
2 answers
1k views

Grouping variables with small sample size

I have a data set with about 50 variables but only 10 observational units. This is due to the underlying science, it's difficult and expensive to increase the sample size. I think a large number of ...
user avatar
  • 412
4 votes
1 answer
2k views

how to perform divisive hierarchical clustering

I've been trying for a long time to figure out how to perform (on paper) the divisive hierarchical clustering algorithem, however I'm not able to understand how to do it exactly. example: I need to ...
user avatar
  • 83
4 votes
2 answers
2k views

Applying Ward's method for calculating linkage

For an assignment, I have used iPython to create the dendrogram below, using Ward's method and Euclidean distance, from the following data: $$a=(0,0)$$ $$b=(1,2)$$ $$c=(3,4)$$ $$d=(4,1)$$ $$e=(2,2)$$ ...
user avatar
  • 61
4 votes
1 answer
54 views

Difference between languages (spoken)?

I'm trying to perform a hierarchical clustering, to aggregate some "zones" or neighborhoods of a city, based on the language that is used most in that zone In order to do so, I have at hand a dataset ...
user avatar
  • 425
4 votes
1 answer
3k views

Text Clustering using TF-IDF and Cosine Similarity

I am attempting to perform hierarchical clustering using (Tf-Idf & cosine distance) on about 25,000 documents that vary in length between 1-3 paragraphs each. With the method above, my question ...
user avatar
  • 43
4 votes
1 answer
2k views

clustering groups but with multiple observations per group

I'd have 10 groups and hundreds of observations per group. In this toy example I only have 3 groups with 20 observations each. I am looking to see if groups are similar so I'm using kmeans to ...
user avatar
4 votes
2 answers
7k views

agglomerative clustering sensitivity to outliers: single-link vs complete-link

Agglomerative clustering can use various measures to calculate distance between two clusters, which is then used to decide which two clusters to merge. Two popular approaches are single-link and ...
user avatar
  • 476
4 votes
2 answers
200 views

Situation that not well represented by hierarchical clustering

The below text is from statistical learning page 394. I highlighted where i stuck. Please help me to understand this. The term hierarchical refers to the fact that clusters obtained by cutting ...
user avatar
  • 510
4 votes
2 answers
937 views

Text mining of machine logs to find correlation between errors in R

I've with me 50 MB data from a machine consisting of event logs such as device status, warning and error. I wish to perform text mining on the same to find correlation between errors i.e. one error ...
user avatar
3 votes
1 answer
419 views

Hierarchical clustering Ward's method. The missing rationale in derivation

The Ward's method is taking distance as how much the sum of squares will increase when we merge them. $d(u,v) = \frac{|u||v|}{|u|+|v|}{|m_u-m_v|}^2$ Please refer to Page 3 of link below. https://...
user avatar
  • 613
3 votes
2 answers
2k views

How to assign existing cluster numbers for future data, using hierarchical clustering algorithms?

Assume we have some good clusters from some clustering algorithm and we want to assign the cluster numbers (labels) to future data (= to enrol new data points into the existing clusters, if to word it ...
user avatar
  • 33.7k
3 votes
2 answers
2k views

Compare two hierarchical trees

I want to measure the similarity between two hierarchical trees generated from the same n objects. The two trees are generated using the same metric at two different instants. Thus, I would like to ...
user avatar
  • 148

1
2 3 4 5
9