0
$\begingroup$

The following content comes from the following site:

[https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/][1]

The algorithm works as follow:

  1. Cluster the observed data, varying the number of clusters from k = 1, …, $ k_{max} $, and compute the corresponding total within intra-cluster variation $ W_k $.

  2. Generate B reference data sets with a random uniform distribution. Cluster each of these reference data sets with varying number of clusters k = 1, …, $ k_{max} $, and compute the corresponding total within intra-cluster variation $ W_{kb} $.

  3. Compute the estimated gap statistic as the deviation of the observed $ W_k $ value from its expected value $ W_{kb} $ under the null hypothesis: $ Gap(k)=\frac{1}{B}\sum{}^B_{b=1}log(W^∗_{kb} )−log(W_k) $. Compute also the standard deviation of the statistics.

  4. Choose the number of clusters as the smallest value of k such that the gap statistic is within one standard deviation of the gap at k+1: $ Gap(k)≥Gap(k + 1)−$ s_k $ + 1 $.

My Questions is:

  • What does "Generate B reference data sets with a random uniform distribution."?
  • How do I generate the B reference data sets based on what?
  • What does the null hypothesis mean?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.