Considering the following sample $X = $ [32.5, 31.7, 29.2, 28.8, 29. , 28.9, 29.9, 30.4, 26.9, 26.5]

I planned to evaluate the hypothesis of this sample having been generated from a normal distribution with an hypothetic $\mu$ and $sigma^2$ trough the use of the Kolmogorov Smirnov test statistic.

My strategy was to solve this problem using bootstrapping, so for each simulation I sampled from $X$ and computed the functional $T(X^*)$ where $T$ is the maximum distance from the empirical distribution of $X^*$ and the theoretical distribution of the distribution I considered. After all the simulations were through I had enough data to plot the empirical distribution of the t statistic, which I am right to call sampling distribution right?!

But when plotting an histogram of this same sampling distribution of the kolmogorov smirnov test statistic I got the following graphic:

enter image description here

My sample size is very low and some discretization was made to the observation, fixed precision. Is this distribution ok and perfectly normal? I am trying to understand if this can happen of (I speculate that it can), or if I messed up during the implementation of the ks-test.

Here's my implementation of the ks-test in R:

$H_0: Sample ~ N(\mu, \sigma^2)$

kstest = function(Xsample, mu, var) {

  n = length(Xsample)
  X_sample = sort(Xsample)

  #List of differences
  diffs = numeric(2*n)

  for(j in 1:n){
    f = pnorm(X_sample[j],mean=mu,sd=sqrt(var))
    diffs[j*2-1] = j/n - f
    diffs[j*2] = f - (j - 1)/n
  • 4
    $\begingroup$ You might have made a coding mistake , possibly there is some statistical error in the design of your code, or maybe your description of what you have done is ambiguous. When I run a quick simulation with small samples ($n=3$ and $n=8$), I get the expected uniform distribution of p-values and a continuous-looking distribution of KS statistics. Unless you provide details of your procedure, we cannot do any more than speculate. $\endgroup$ – whuber Nov 1 '16 at 13:47
  • $\begingroup$ Thank you for your feedback, I updated my question. I am seriously considering that the problem is lack of precision in my samples.. or else there is something I am missing with the process. $\endgroup$ – Ramalho Nov 1 '16 at 15:09
  • $\begingroup$ There appear to be some issues with your implementation of kstest. Compare it to the output of stats::ks.test. (This will not affect the shape of your histogram much, though.) $\endgroup$ – whuber Nov 1 '16 at 16:34
  • $\begingroup$ Thank you!! I got it corrected now but the behaviour/shape is the same as you have suggested. Do you have any idea of what is causing it? Might it be the small sample? fixed precision? Can the sampling distribution assume different shapes other than bell curves? We know that, that is the expected behaviour when the statistic is the mean (CLT), but does the same thing need to apply to other statistics? Thank you a lot! $\endgroup$ – Ramalho Nov 1 '16 at 20:28
  • $\begingroup$ Even the bootstrap distribution of the mean can exhibit severe multimodality, especially for small datasets. You might find it illuminating to make some normal QQ plots for a few bootstrap samples. $\endgroup$ – whuber Nov 1 '16 at 21:26

As @whuber suggested the bootstrap distribution, sampling distribution, can exhibit strong multimodality and points of discontinuity. For the sample mean there are only $C^{2N-1}_{N-1}$ possible values that $\bar{X}^*$ can take, this means that such a small sample as mine will have a very discrete behaviour like above. Fortunately as $N \rightarrow \infty$ the sampling distribution becomes more continuous.

I solved the problem by using the parametric bootstrap instead of the non parametric strategy I was using before. That is, I assume that X is normally distributed and estimate the distribution parameters through maximum likelihood. Then at each bootstrap I sample from the estimated distribution and apply the kstest to that sample. The sampling distribution becomes much more well behaved using this alternate strategy:

enter image description here

This really helped me understand concepts such as sampling distribution, parametric and non parametric bootstrap much better. For reference, in cases such that no prior distribution can be assumed and we are forced to use a non parametric bootstrap then we can also use a smoothed bootstrap to force a more or less well behaved sampling distribution.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.