12
$\begingroup$

It is well known that it is problematic to choose a statistical test based on the outcome of another statistical test, as the p-values are difficult to impossible to interpret (e.g. Choosing a statistical test based on the outcome of another (e.g. normality)). However, this is still standard practice in many applications and usually does not seem to be noticed or discussed in applied papers. Looking through the literature, I failed to find a paper that actually discusses this phenomenon.

I would appreciate links to any publications relating to choosing a statistical test based on the outcome of another statistical test, especially any that are accessible to applied scientists.

$\endgroup$
5
  • $\begingroup$ Unrelated comment: In my search, I stumbled across a paper by RS Nickerson 'Null hypothesis significance testing: a review of an old and continuing controversy', which does not discuss this particular phenomenon, but also seemed nice to give to applied scientists. $\endgroup$
    – Rob Hall
    Commented Oct 9, 2013 at 12:54
  • 2
    $\begingroup$ Long ago I posted a detailed analysis of one such situation at quantdec.com/envstats/notes/class_12/ucl.htm: it studies the properties of a UCL that is determined by a procedure chosen conditional on the results of a preliminary hypothesis test (concerning the underlying distribution). $\endgroup$
    – whuber
    Commented Oct 10, 2013 at 16:50
  • 3
    $\begingroup$ This one might interest you (also see this). Then there's this. That relates to testing equality of variance and testing normality respectively before a two-sample t-test. $\endgroup$
    – Glen_b
    Commented Oct 11, 2013 at 5:19
  • $\begingroup$ It seems to me that to simply say that "It is well known that it is problematic" is to provide insufficient specificity because the problematic nature probably depends on the statistical framework within which one is working. Problems for frequentist interpretation may not be problems for methods that assess the evidential meaning of data. $\endgroup$ Commented Dec 7, 2013 at 6:47
  • $\begingroup$ Maybe a simple example where this is problematic would serve much the same purpose as a citation. $\endgroup$
    – BKay
    Commented Oct 7, 2014 at 15:40

1 Answer 1

2
$\begingroup$

I think that the following research paper on chain procedures is relevant to and might be helpful in answering your question: http://www.multxpert.com/doc/md2011.pdf.

$\endgroup$
2
  • 1
    $\begingroup$ Thank you for the interesting paper. If I understand it correctly, it looks at data-driven allocation of alpha to an ordered set of hypotheses. Such a procedure could then simply add all variants of a hypothesis to a list (e.g. the hypothesis given that a paremetric test can be used and the hypothesis given that there is some evidence that a non-paremetric test should be used). While this should a be sensible approach in a Neyman–Pearson framework, I am not sure this solves the problem of interpreting the p-values in the sense of Fisher. $\endgroup$
    – Rob Hall
    Commented Nov 26, 2014 at 0:26
  • 1
    $\begingroup$ @RobHall: You're very welcome! Frankly, I browsed the paper without delving into details, so at the present time I can't really make a worthwhile comment. But I hope to review this paper when I will have a chance. By the way, here is another interesting paper that might be relevant to this, where author argues that the two frameworks can be considered complementary: stat.duke.edu/courses/Spring07/sta215/Ref/Lehm1993.pdf. $\endgroup$ Commented Nov 26, 2014 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.