Clustering inertia formula in scikit learn I would like to code a kmeans clustering in python using pandas and scikit learn. In order to select the good k, I would like to code the Gap Statistic from Tibshirani and al 2001 (pdf).
I would like to know if I could use inertia_ result from scikit and adapt the gap statistic formula without having to recode all the distances calculation.  
Does anyone know the inertia formula used in scikit / know an easy way to recode the gap statistic using high level distance functions? 
 A: I guess I found my answer for kmeans clustering:  
By looking at the git source code, I found that for scikit learn, inertia is calculated as the sum of squared distance for each point to it's closest centroid, i.e., its assigned cluster. So $I = \sum_{i}(d(i,cr))$ where $cr$ is the centroid of the assigned cluster and $d$ is the squared distance.  
Now the formula of gap statistic involves 
$$
W_k = \sum_{r=1}^{k}\frac 1 {(2*n_r) }D_r
$$ 
where $D_r$ is the sum of the squared distances between all points in cluster $r$.
By introducing $+c$, $-c$ in the squared distance formula ($c$ being the centroid of cluster $r$ coordinates), I have a term that corresponds to Inertia (as in scikit) + a term that disappears if each $c$ is the barycentre of each cluster (which it is supposed to be in kmeans). So I guess $W_k$ is in fact scikit Inertia. 
I have still two questions:


*

*Do you think my calculus is correct? (For example, I don't know if it holds for hierarchical clustering.) 

*If I am correct above, I have coded the gap statistic (as difference of log inertias between estimation and clustering) and it performs badly especially on the iris dataset, has anyone tried it?

