2
$\begingroup$

This question is based on a comment by John Kruschke in his BEST paper, pages 589-590.

Kruschke, John K. "Bayesian estimation supersedes the t test." Journal of Experimental Psychology: General 142.2 (2013): 573.

Consider...a case in which there is a windfall of data, perhaps caused by miscommunication so two research assistants collect data instead of only one. That is, the researcher intended to collect $N = 8$ per group, but the miscommunication produced $N = 16$ per group. Most analysts and all statistical software would use $N = 16$ per group to compute a p value. This is inappropriate, because the space of possible $t_{null}$ values from the null hypothesis should actually be dominated by the intended sampling scheme, not by a rare accidental quirk.

Why should we take $N=8$ instead of $N=16?$ Yes, I see that Kruschke says "the space of possible $t_{null}$ values from the null hypothesis should actually be dominated by the intended sampling scheme, not by a rare accidental quirk," but why?

$\endgroup$
2
$\begingroup$

Every simulated sample in the sampling distribution is supposed to be generated in the same way as the actual sample. That's the whole point of a sampling distribution: To represent what would have happened if the data were generated the same way as the actual sample but from a hypothetical world (e.g., null hypothesis). Therefore, if the actual data had a random sample size (not a fixed sample size), the randomness of the sample size should be incorporated into simulated samples. The resulting sampling distribution can be thought of as a probabilistic mixture of fixed-$N$ sampling distributions, with each $N$ weighted by the probability that it happens.

You can read a little more in an article (I wrote); e.g., "It is important to understand that the $p$ value would be different if the stopping or testing intentions were different. For example, if data collection were stopped because time ran out instead of because $N$ reached 18, then the sample size would be a random number and the sampling distribution would be different, hence the $p$ value would be different. When $N$ is a random value, the sampling distribution is a probabilistic mixture of different fixed-$N$ sampling distributions, and the resulting mixture is (in general) not the same as any one of the fixed-$N$ distributions."

I should point out that my perspective on this issue is unconventional, but I think also appropriate (otherwise I wouldn't promote it :-). For extended numerical examples of random-$N$ sampling distributions, see Ch 11 of DBDA2E, and for a rebuttal of arguments than random-$N$ sampling can be treated as if it were fixed-$N$, see p. 313 of DBDA2E.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I’ve got some reading and thinking to do, but what a cool response to get! $\endgroup$ – Dave Oct 18 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.