I was going through an explanation of gap statistic here when I happened to come across the phrase null reference distribution of the data. And this is something I need help to understand.

From what I understand from reading online that to conduct a hypothesis testing, you'll need data points to compare against the actual data and these points are provided by a particular distribution of data. Using these two a statistical test will be conducted which will provide us with some information about the actual data we have.

Am I understanding this correctly? If so, then how do we select this null reference distribution of data? The actual distribution could be anything. It could be a combination of multiple altogether different distributions. How do we select one then?

Specifically speaking, in this case, while explaining the GAP statistic, the documentation says

Generate B reference data sets with a random uniform distribution

How did they come to conclude that they should use a random uniform distribution? Why not Gaussian or some other distribution?

And also, why do I need B copies. Is it a sense cross validation-esque?


1 Answer 1


Null reference distribution of data refers to generating random samples of testing data under null hypothesis. When we say Random sample, it means the test statistics of null distribution data is often unknown. There are many resampling methods that can provide consistent estimators for null distribution such as model-based bootstrap method. One should be careful while choosing the estimator for null distribution because improper choice of null distribution has significant effect on type-1 error. For more information about null distribution you can refer this link.

In the standard Gap statistics method, we standardize the graph of log($W_k$), where $W_k$ is within cluster dispersion, by comparing it to the expectation of $W_k$ under appropriate null reference distribution of the data. In statistics, not every algorithm is suitable for all the cases. Every algorithm has its drawback. It is not mandatory to use random uniform distribution of data in all the cases. In particular, the choice of choosing a random uniform distribution over Normal distribution solely depends upon the compactness of cluster your model has generated. It has been studied by Yin et al. (2008) that in situation where clusters contain data points of different densities, standard Gap method might fail and he suggested to use reference data sets sampled from Normal distribution rather than Uniform distribution. For more information, you can refer this link.

Intuitively, choosing random sample over uniform distribution takes care of biases present within the dataset. we choose uniform distribution just to make sure that the probability of occurrences of one variable should not affect the other as in uniform distribution, all the data sample generated have equal probabilities of occurrence and overall, it does not affect the null distribution of generated sample.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.