4
$\begingroup$

I need to use gap statistics in my research, and I was drove to read the classic papaer by Tibshirani et. al, 2001 (here is the paper). I am confused with two points. First, why does it substract one standard deviation when obtaining the optimal K using gap statistic for clustering analysis?

$$ Gap(k)\geqslant Gap(k+1)-s_k, s_k=std_k \times \sqrt {1+\frac{1}{B}} $$

Second, why is the standard deviation calculated in this way? I only know that the variance of sampling distribution has a relation with the variance of the population by $\sigma_{sam}^2=\frac {1}{n} \times \sigma_p^2$

$\endgroup$
0

2 Answers 2

1
$\begingroup$

Subtracting the standard deviation is a test whether the increase in gap statistic is statistically significant.

Variance of the mean of the sample is equal to the variance of the sample divided by the sample size. But the relation between the variance of the sample and the variance of the population is different, and the factor 1 + 1 / B probably accounts for that.

$\endgroup$
0
$\begingroup$

Maybe I misunderstood their point, but contrary to the answer by @quant_dev, I don't think the factor of $\sqrt {1+\frac{1}{B}}$ accounts for the sample vs population variance correction (Bessel's correction), which addresses the bias in the estimate of the population variance from a sample by using $n-1$ rather than $n$. This correction specifically accounts for the degrees of freedom in estimating variance from a finite sample size.

Instead, $\sqrt {1+\frac{1}{B}}$ attempts to adjust the standard deviation of the Gap statistics across bootstrap samples to account for the limited number of samples and the inherent randomness in bootstrapping. Thus, while there is no direct connection between the two adjustments, both attempt to make the methods more robust.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.