21
$\begingroup$

I'm reading John Kruschke's "Doing Bayesian Data Analysis" slides, but actually have a question about his interpretation of t-tests and/or the whole null-hypothesis significance testing framework. He argues that p-values are ill-defined because they depend on the investigator's intentions.

In particular, he gives an example (pages 3-6) of two labs that collect identical data sets comparing two treatments. One lab commits to collect data from 12 subjects (6 per condition), while the other collects data for a fixed duration, which also happens to yield 12 subjects. According to the slides, the critical $t$-value for $p<0.05$ differs between these two data collection schemes: $t_{\textrm{crit}}=2.33$ for the former, but $t_{\textrm{crit}}=2.45$ for the latter!

A blog post--which I now cannot find--suggested that the fixed-duration scenario has more degrees of freedom since they could have collected data from 11, 13, or any other number of subjects, while the fixed-N scenario, by definition, has $N=12$.

Could someone please explain to me:

  • Why the critical value would differ between these conditions?

  • (Assuming it's an issue) How one would go about correcting/comparing for the effects of different stopping criteria?

I know that setting the stopping criteria based on significance (e.g., sample until $p<0.05$) can inflate the chances of a Type I error, but that doesn't seem to be going on here, since neither stopping rule depends on the outcome of the analysis.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
11
$\begingroup$

Here's some more info: http://doingbayesiandataanalysis.blogspot.com/2012/07/sampling-distributions-of-t-when.html

A more complete discussion is provided here: http://www.indiana.edu/~kruschke/BEST/ That article considers p values for stopping at threshold N, stopping at threshold duration, and stopping at threshold t value.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Wow! Straight from the horse's mouth, as it were...It's definitely an interesting idea that hadn't occurred to me. Thanks for the additional info. $\endgroup$
    – Matt Krause
    Jul 21, 2012 at 17:35
  • $\begingroup$ I wanted to add that this is discussed at length in Dr. Kruschke's book (in Chapter 11). $\endgroup$
    – Matt Krause
    Jun 23, 2015 at 15:08
13
$\begingroup$

I finally tracked down the paper associated with the slides: Kruschke (2010), also available directly from the author (via CiteSeerX) here, since the journal is not widely carried. The explanation is a little bit prosaic, but I'm still not sure I buy it.

In the fixed-N case, the critical $t$-value is computed as follows: $2N$ samples are randomly drawn from the (same) population and a $t$-value is calculated. This process is repeated many times to build up a null distribution. Finally, $t_{crit}$ is set to be the 95th percentile of that distribution.

For the fixed duration case, he assumes that subjects arrive at a mean rate $\lambda$. The null distribution is constructed by repeating two steps. In the first step, the number of subjects for each condition $N_1$ and $N_2$ is drawn from a possion distribution with parameter $\lambda$. Next, $N_1$ and $N_2$ random draws from the population are used to calculate a $t$-value. This is repeated many times, and $t_{crit}$ is set to be the 95th percentile of that distribution.

This seems a little...cheeky...to me. As I understand it, there isn't a single $t$-distribution; instead it's a family of distributions, with a shape partly determined by the degrees-of-freedom parameter. For the fixed-$N$ condition, there are $N$ subjects per group and the appropriate $t$-value for an unpaired t-test is the one with $2N-2$ degrees of freedom, which is presumably what his simulation reproduces.

In the other condition, it seems like the "$t$"-like distribution is actually a combination of samples from many different $t$-distributions, depending on the specific draws. By setting $\lambda=N$, one could get the average degrees of freedom to equal $2N-N$, but that's not quite enough. For example, the average of the $t$-distributions for $\nu=1$ and $\nu=5$ doesn't seem to be the $t$-distribution with 3 degrees of freedom.

In summary:

  • The author was generating $t_{crit}$ by simulation, instead of just calculating them from the CDF.
  • The way the author simulated the fixed-duration scenario seems like it might fatten up the tails of the corresponding $t$-distribution.
  • I remain unconvinced that this is actually a problem, but would be happy to read/upvote/accept answers if anyone thinks otherwise.
Improve this answer
$\endgroup$
5
  • $\begingroup$ Why are you able to answer your own question and give it a check mark? Doesn't seem like you should be able to give yourself rep pointe! $\endgroup$
    – Michael R. Chernick
    Jul 20, 2012 at 0:36
  • 5
    $\begingroup$ There's nothing wrong to answer his own question, Michael. $\endgroup$
    – chl
    Jul 20, 2012 at 9:18
  • $\begingroup$ @MichaelChernick, I believe you don't get any rep if you accept your own answer. At the time, it seemed like the right thing to do, since I had more or less tracked down the answer in the intervening two weeks, but I've switched my accept to John K. Kruschke's answer since he's clearly the authority on his own slides :-) $\endgroup$
    – Matt Krause
    Jul 21, 2012 at 17:24
  • $\begingroup$ Interesting thanks. But I don't see why one should check their own answer at any time even if it appears to be correct and the best. We have established that checking your own answer doesn't give you rep points. $\endgroup$
    – Michael R. Chernick
    Jul 21, 2012 at 17:45
  • 3
    $\begingroup$ Since marking an answer as accepted has no other purpose than to indicate a correct solution (for future visitors), especially where no other has been proposed, I see no problem with that. Personally, I have upvoted this answer long ago, because I appreciate that the OP lets us benefit from its own research. And I'm really sorry for not being able to give an additional vote for the simple fact of following this thread and updating his decision. PS "We have established..." refers to Why is it possible to give yourself reputation points?. $\endgroup$
    – chl
    Jul 21, 2012 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.