How much variation should a clustering algorithm explain? When running a cluster analysis, the algorithm used normally returns a measure of how much variation the clustering explains. e.g.
"This clustering explains 96 % of the variation in the data"
However, I am not sure how much of the variation a clustering method should explain in order to be "good enough", or how little variance explained is basis for rejecting the clustering and trying another approach (e.g. a different amount of clusters).
Any general rules on that?
Thanks!
 A: The comment provided by ttnphns is correct; the percent variation accounted for is so closely tied to the variability of the data that a hard-and-fast rule simply does not exist.  Additionally, adding additional clusters will always increase the % variance accounted for, so what you really need to be asking is something along the lines of: is the % increase in variance accounted for worth adding another cluster (increasing complexity).
One approach is to try some number of clusters, calculate the % variance accounted for at each step, and then try to determine some threshold for change in variance beyond which adding clusters had little yield.  For example:
set.seed(1)
z = data.frame(x=1:200,y=c(runif(100,0,1),runif(100,1,2)))
p = numeric()

for (i in 1:4){
   K= kmeans(z,i)
   p[i] = 1 - K$tot.withinss / K$totss
}

plot(p,xlab='# of clusters',ylab='% variance accounted',type='o')


A: The clustering where every point is its own cluster explains 100% of the variance, doesn't it?
Is that a useful result? No.
Don't use this measure as a criterion. It's monotone increasing with the number of clusters.
Instead evalute the clustering by your task: does it help solving your actual problem?
In the end, you need a helpful clustering, not one that minimizes/maximizes some mathematical property, don't you? Or is the mathematical property your problem?
