degrees of freedom in a t-test in Kruschke's BEST paper

Question

This question is based on a comment by John Kruschke in his BEST paper, pages 589-590.

Kruschke, John K. "Bayesian estimation supersedes the t test." Journal of Experimental Psychology: General 142.2 (2013): 573.

Consider...a case in which there is a windfall of data, perhaps caused by miscommunication so two research assistants collect data instead of only one. That is, the researcher intended to collect $N = 8$ per group, but the miscommunication produced $N = 16$ per group. Most analysts and all statistical software would use $N = 16$ per group to compute a p value. This is inappropriate, because the space of possible $t_{null}$ values from the null hypothesis should actually be dominated by the intended sampling scheme, not by a rare accidental quirk.

Why should we take $N=8$ instead of $N=16?$ Yes, I see that Kruschke says "the space of possible $t_{null}$ values from the null hypothesis should actually be dominated by the intended sampling scheme, not by a rare accidental quirk," but why?

Answer 1

Every simulated sample in the sampling distribution is supposed to be generated in the same way as the actual sample. That's the whole point of a sampling distribution: To represent what would have happened if the data were generated the same way as the actual sample but from a hypothetical world (e.g., null hypothesis). Therefore, if the actual data had a random sample size (not a fixed sample size), the randomness of the sample size should be incorporated into simulated samples. The resulting sampling distribution can be thought of as a probabilistic mixture of fixed-$N$ sampling distributions, with each $N$ weighted by the probability that it happens.

You can read a little more in an article (I wrote); e.g., "It is important to understand that the $p$ value would be different if the stopping or testing intentions were different. For example, if data collection were stopped because time ran out instead of because $N$ reached 18, then the sample size would be a random number and the sampling distribution would be different, hence the $p$ value would be different. When $N$ is a random value, the sampling distribution is a probabilistic mixture of different fixed-$N$ sampling distributions, and the resulting mixture is (in general) not the same as any one of the fixed-$N$ distributions."

I should point out that my perspective on this issue is unconventional, but I think also appropriate (otherwise I wouldn't promote it :-). For extended numerical examples of random-$N$ sampling distributions, see Ch 11 of DBDA2E, and for a rebuttal of arguments than random-$N$ sampling can be treated as if it were fixed-$N$, see p. 313 of DBDA2E.