I need to use gap statistics in my research, and I was drove to read the classic papaer by Tibshirani et. al, 2001 (here is the paper). I am confused with two points. First, why does it substract one standard deviation when obtaining the optimal K using gap statistic for clustering analysis?

$$ Gap(k)\geqslant Gap(k+1)-s_k, s_k=std_k \times \sqrt {1+\frac{1}{B}} $$

Second, why is the standard deviation calculated in this way? I only know that the variance of sampling distribution has a relation with the variance of the population by $\sigma_{sam}^2=\frac {1}{n} \times \sigma_p^2$


Subtracting the standard deviation is a test whether the increase in gap statistic is statistically significant.

Variance of the mean of the sample is equal to the variance of the sample divided by the sample size. But the relation between the variance of the sample and the variance of the population is different, and the factor 1 + 1 / B probably accounts for that.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.