16
votes
Clustering -- Intuition behind Kleinberg's Impossibility Theorem
One way or another, every clustering algorithm relies on some notion of “proximity” of points. It seems intuitively clear that you can either use a relative (scale-invariant) notion or an absolute (...
12
votes
Accepted
Are there algorithms to cluster Graphs, not just cluster nodes in a graph?
The main problem here seems to me to be about defining and finding the (dis)similarity between different graphs.
The 'graph edit distance' (defining distance in terms of number of operations ...
- 86.4k
9
votes
Accepted
Best practices in the selection of distance metric and clustering methods for gene expression data
This will probably not be the answer you want or expect, but this is how I see these things.
Clustering problem
Clustering, to a degree, is almost always a subjective procedure. You decide how you ...
- 5,114
8
votes
Does k-means have any advantages over HDBSCAN expect for runtime?
Randomization can be valuable. You can run k-means several times to get different possible clusters, as not all may be good. With HDBSCAN, you will always get the same result again.
Classifier: k-...
- 42.9k
7
votes
Can sub-optimality of various hierarchical clustering methods be assessed or ranked?
Only single-linkage is optimal. Complete-linkage fails, so your intuition was wrong there. ;-)
As a simple example, consider the one-dimensional data set 1,2,3,4,5,6.
The (unique) optimum solution ...
- 3,398
7
votes
Accepted
Conditional intraclass correlation (ICC) from a linear mixed model as evidence for test-retest reliability?
Yes, you can do this and interpret it as you think. I have read about such an interpretation in the second chapter of Sophia Rabe-Hesketh and Anders Skrondal's Multilevel and Longitudinal Modeling ...
- 5,880
6
votes
Hierarchical clustering with categorical variables
Yes of course, categorical data are frequently a subject of cluster
analysis, especially hierarchical. A lot of proximity measures exist
for binary variables (including dummy sets which are the ...
6
votes
Choosing the right linkage method for hierarchical clustering
The correlation between the distance matrix and the cophenetic distance is one metric to help assess which clustering linkage to select. From ?cophenetic:
It can ...
- 310
6
votes
Survey design for multilevel models
Executive summary: this is much harder than you would expect, and there is neither a standard implementation, nor even an accepted estimator.
Let's fix some terminology. In a survey, you have ...
- 46.8k
6
votes
Accepted
Question about Silhouette index calculation using scikit
You can use any dissimilarity measure (or distance metric) that compares pairs of points, as described in [1,2 3]. It makes sense to use the same measure as used to construct the clusters as then they ...
- 456
5
votes
Accepted
Motivation for Ward's definition of error sum of squares (ESS)
\begin{align}
\operatorname{Var}(\vec x) \propto \sum_{i=1}^n(x_i - \bar x)^2 &= \sum_i x_i^2 - 2\bar x \sum_ix_i + n \bar x^2 \\
&= \sum_i x_i^2 - n \bar x^2 = \sum_i x_i^2 - \frac 1n \left(\...
- 20.8k
5
votes
Accepted
When doing hierarchical clustering, do we need to exclude variables with high correlation?
Ultimately the answer is "it depends". It depends on various things, including potential preprocessing and the distance you use (I guess Euclidean but be aware that this is not the only ...
- 28.3k
5
votes
Question about Silhouette index calculation using scikit
(1) The distance function used should express as "correctly" as possible what distance "means" in the application in question. Any calculation based on distances, be it the ...
- 28.3k
4
votes
how to perform divisive hierarchical clustering
This question is rather old, but I think an answer may help some people.
I understand that you mean by this 5 points in three dimensions.
You say "the divisive hierarchical clustering algorithm". I ...
- 2,650
4
votes
Accepted
K-Means Cluster has over 50% of the points in one cluster. How to optimize it?
This is very typical behavior of k-means when applies to non-continuous data. It's not what k-means is designed for, you are essentially operating it out of its specifications. Also, k-means is very ...
- 42.9k
4
votes
Accepted
Pooling levels of categorical variables for regression trees
I have implemented my solution to this. I wrote two functions:
prox_matrix(df, target, features, cluster_dimension,trees = 10)
Parameters
df: Input dataframe
target: Dependant variable you are ...
- 521
4
votes
Accepted
Hierarchical clustering Ward's method. The missing rationale in derivation
Rather than reverse engineering the code, also check for references and literature. These algorithms often long predate sklearn.
Even Wikipedia has this equation, known as Lance Williams equations: ...
- 42.9k
4
votes
K means clustering breakup---galaxy spectrum data set
In many real datasets (obviously I don't know about yours), clusters are not well separated, and even if they are, $k$-means clusters will not necessarily correspond to well separated subsets of the ...
- 28.3k
3
votes
Applying Ward's method for calculating linkage
Having banged my head on the wall for the last 2 hours on this, I feel your pain.
The result is the square root of the increase in within-cluster sum of squares (vs. cluster means), multiplied by $\...
- 31
3
votes
Accepted
"Updating" hierarchical clustering
Hierarchical clustering results are not very well updateable.
If the nearest neighbor of a new point (similar for a disappearing) point is at height h, then you should be able to keep anything below ...
- 42.9k
3
votes
K-Means Cluster has over 50% of the points in one cluster. How to optimize it?
I am using K means clustering on the "words" matrix from an SVD of a Tf Idf matrix and got similar results. I found the sum of the squares of the features for this large cluster and found they were ...
- 31
3
votes
Testing whether two datasets cluster similarly
One problem with your scenario, which is unrelated to clustering, is that you want to check whether your results are similar, and it is always harder to do than testing whether they are different. ...
- 178
3
votes
How to assign existing cluster numbers for future data, using hierarchical clustering algorithms?
Don't do this.
It's not even a good idea for k-means.
Do not assume the clustering is perfect, and also do not assume that it never changes. And you won't be able to react appropriately.
For ...
- 42.9k
3
votes
How to assign existing cluster numbers for future data, using hierarchical clustering algorithms?
My answer is about agglomerative (i.e. bottom-up) hierarchical cluster analysis, HAC, which methods are overviewed here. (I'm not quite sure if the answer can be extended to divisive, top-down ...
- 58.8k
3
votes
Choosing the number of clusters - clustering validation criterions vs domain theoretical considerations
The keys are finding meaningful clusters and what you value
in the resulting clusters.
Let me illustrate with a simple example.
The example is two Gaussian clusters that are pretty well separated. ...
- 2,650
3
votes
Accepted
Comparison between hierarchical clustering and principal component analysis (PCA)
There is a very weak link because both PCA and k-means clustering try to minimize the least squared deviations. But that is a pretty much universal principle, and there exists so much more clustering ...
- 42.9k
3
votes
What are possible reasons of clustering failure
The main reason is inappropriate data preprocessing.
People tend to assume they can just dump the data into a black box algorithm and get out clusters. That does not work. Because clustering is ...
- 42.9k
3
votes
Cluster analysis of variables or observations?
From a mathematical point of view, a standard dataset is just a matrix of numbers organized into rows and columns. We attach meanings to these, and think of the rows as pertaining to patients and the ...
3
votes
Accepted
Clustering for medium data
Frequently faced issues while clustering are,
1. full distance matrix,
2. Memory,
3. runtime.
In hierarchical clustering, we create hierarchical decomposition of the given set of data in two ways ...
- 146
3
votes
Accepted
Bayesian Hierarchical Clustering: How to calculate probability of Data under $H_1$?
Hope this isn't too late to help! In short, yes, you have what I believe is the right idea.
I've been messing around with this a little bit myself, and what's being referenced here is the fact that ...
- 46
Only top scored, non community-wiki answers of a minimum length are eligible