3
$\begingroup$

Let's say I have a fixed set of data containing one variable. I'm interested in whether the variable shows a difference between two conditions. Now I perform a test and determine a p-value somewhere above my desired alpha. Being disappointed, I iterate over multiple different alternatives of tests for the same hypothesis and find that one test gives a p-value below my alpha.

This seems to be some kind of "fishing expedition", but not of the kind that a Bonferroni correction is supposed to fix: I'm always testing the same hypothesis. Still, somehow my gut feeling tells me this is fishy. What is generally considered appropriate here? Do the fishing and report it in the paper? Go with the first p-value? Calculate the mean p?

$\endgroup$
1
  • 2
    $\begingroup$ You are right that this is widely regarded as dubious, but how dubious? You would not fault yourself for trying different hammers or screwdrivers before deciding on the best for a job. How far is this similar? On the strictest interpretation, one should specify the tests to used in advance and use only those tests. That's admirably puritan, but I'd guess it's minority practice. In what I read, I often think about a test that has been used as just another kind of exploratory data analysis, especially when I would tend to favour a different test or no test or all (e.g. a graphical approach). $\endgroup$
    – Nick Cox
    Jul 5, 2013 at 11:48

2 Answers 2

6
$\begingroup$

Often these different tests test subtly different hypotheses. For example, many tests that are used to check for normality/Gaussianity of a distribution actually look at very specific deviations from normality (e.g. no skewness) and tests differ with respect to what quantity they exactly look at. So I would start with finding out what the difference is between these tests and decide which hypothesis is the one that is really of interest to you.

If the different tests really test the same null hypothesis and result in different $p$-values, that it is an issue of power. Typically, there is a literature comparing these tests, and you can just look up which test best applies to your situation.

Having said that, my first guess would be that your different tests actually test similar but not the same hypotheses. In general, once you understand where the differences between tests come from, the choice becomes pretty obvious.

$\endgroup$
1
  • 1
    $\begingroup$ Re 2nd paragraph: two different tests can be of the same null hypothesis but have different power against different alternatives. Even if you're not a Puritan, you can't allow yourself too much latitude in choosing/devising a test against an alternative suggested by looking at the data. So I think that's wrong. $\endgroup$ Jul 5, 2013 at 12:11
3
$\begingroup$

There's no need to assume that different null hypotheses are being tested to use the Bonferroni correction. (See Cox & Hinkley (1974), Theoretical Statistics, 3.4 (iii) for a derivation & discussion of its application to multiple tests of the same null hypothesis—your question involves optional stopping rather than choosing the lowest p-value for a fixed no. tests, but I'm assuming that wasn't the important point.) If you really were going to keep on performing arbitrary tests until you get a p-value you like, a multiple-testing correction would seem more relevant than it usually does. Bear in mind that any sample is extreme in some respect, so you can always find a test that will reject the null hypothesis at an $\alpha$ of your choosing. In practice using a test statistic that's obviously post hoc or outré will excite suspicion (see Witztum et al. (1994), Equidistant Letter Sequences in the Book of Genesis, Statistical Science, 9, 3).

As to what to report: there is a distinction between exploratory & confirmatory analyses. Real analyses often have elements of both. Best to simply tell the truth about what you did, & let your readers decide for themselves whether they're bothered or not by multiple testing.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.