John Kruschke has written widely on the misleading information provided by p values. I understand nearly everything he says, although there is one aspect I am a little confused about.

Kruschke writes in this article that the p value is sensitive to sampling interruptions or windfall. For instance, if you intended to collect data from 40 subjects, but had to spend time fixing some equipment, and only ended up collecting data from 20 subjects, the resulting analysis would be inappropriate. Alternatively, if you end up collecting more than 40 subjects. Kruschke explains this is the case because the sampling distribution under the null hypothesis depends on the intended sample size…

To me (and I know I am not understanding his argument), regardless of the eventual sample size collected, the actual distribution of test statistics under the null hypothesis are always going to be calculated based on what data we put into the computer, i.e. the final sample size we end up with. Our 'intentions' cannot influence what goes into the computer, and thus not influence the underlying, hypothetical null distribution of values.

Is Kruschke saying that, by intending to collect a fixed N-sample size, we are hypothetically suggesting a null distribution we believe is suitable for our experiment. But by interrupting our experiment and collecting larger sample sizes, our beliefs in the analysis and the resulting p value necessarily changes?

I hope someone can make this issue clearer for me. I understand his other points, e.g. the difference between a fixed N-sample size and a fixed sample duration, since our assumptions are directly given to the computer (e.g. the sampling distributions mechanically change in the computer). But the interruption/windfall example I am struggling with.

Thanks in advance!


The distribution of the test statistic under the null hypothesis has nothing to do with the data you put into the computer. It is completely defined by the test statistic (which is just a mathematical function) and the assumed distribution of samples under the null hypothesis. If your assumed distribution of samples has 40 subjects every time, then you will get one distribution. If your assumed distribution of samples has a varying number of subjects, then you will get a different distribution. In both cases, the statistic you get from your data is the same, but the p-value is different. This is explained at the beginning of the section "A p value depends on sampling intentions".

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.