John Kruschke has written widely on the misleading information provided by p values. I understand nearly everything he says, although there is one aspect I am a little confused about.
Kruschke writes in this article that the p value is sensitive to sampling interruptions or windfall. For instance, if you intended to collect data from 40 subjects, but had to spend time fixing some equipment, and only ended up collecting data from 20 subjects, the resulting analysis would be inappropriate. Alternatively, if you end up collecting more than 40 subjects. Kruschke explains this is the case because the sampling distribution under the null hypothesis depends on the intended sample size…
To me (and I know I am not understanding his argument), regardless of the eventual sample size collected, the actual distribution of test statistics under the null hypothesis are always going to be calculated based on what data we put into the computer, i.e. the final sample size we end up with. Our 'intentions' cannot influence what goes into the computer, and thus not influence the underlying, hypothetical null distribution of values.
Is Kruschke saying that, by intending to collect a fixed N-sample size, we are hypothetically suggesting a null distribution we believe is suitable for our experiment. But by interrupting our experiment and collecting larger sample sizes, our beliefs in the analysis and the resulting p value necessarily changes?
I hope someone can make this issue clearer for me. I understand his other points, e.g. the difference between a fixed N-sample size and a fixed sample duration, since our assumptions are directly given to the computer (e.g. the sampling distributions mechanically change in the computer). But the interruption/windfall example I am struggling with.
Thanks in advance!
