# degrees of freedom in a t-test in Kruschke's BEST paper

This question is based on a comment by John Kruschke in his BEST paper, pages 589-590.

Kruschke, John K. "Bayesian estimation supersedes the t test." Journal of Experimental Psychology: General 142.2 (2013): 573.

Consider...a case in which there is a windfall of data, perhaps caused by miscommunication so two research assistants collect data instead of only one. That is, the researcher intended to collect $$N = 8$$ per group, but the miscommunication produced $$N = 16$$ per group. Most analysts and all statistical software would use $$N = 16$$ per group to compute a p value. This is inappropriate, because the space of possible $$t_{null}$$ values from the null hypothesis should actually be dominated by the intended sampling scheme, not by a rare accidental quirk.

Why should we take $$N=8$$ instead of $$N=16?$$ Yes, I see that Kruschke says "the space of possible $$t_{null}$$ values from the null hypothesis should actually be dominated by the intended sampling scheme, not by a rare accidental quirk," but why?

Every simulated sample in the sampling distribution is supposed to be generated in the same way as the actual sample. That's the whole point of a sampling distribution: To represent what would have happened if the data were generated the same way as the actual sample but from a hypothetical world (e.g., null hypothesis). Therefore, if the actual data had a random sample size (not a fixed sample size), the randomness of the sample size should be incorporated into simulated samples. The resulting sampling distribution can be thought of as a probabilistic mixture of fixed-$$N$$ sampling distributions, with each $$N$$ weighted by the probability that it happens.
You can read a little more in an article (I wrote); e.g., "It is important to understand that the $$p$$ value would be different if the stopping or testing intentions were different. For example, if data collection were stopped because time ran out instead of because $$N$$ reached 18, then the sample size would be a random number and the sampling distribution would be different, hence the $$p$$ value would be different. When $$N$$ is a random value, the sampling distribution is a probabilistic mixture of different fixed-$$N$$ sampling distributions, and the resulting mixture is (in general) not the same as any one of the fixed-$$N$$ distributions."
I should point out that my perspective on this issue is unconventional, but I think also appropriate (otherwise I wouldn't promote it :-). For extended numerical examples of random-$$N$$ sampling distributions, see Ch 11 of DBDA2E, and for a rebuttal of arguments than random-$$N$$ sampling can be treated as if it were fixed-$$N$$, see p. 313 of DBDA2E.