# Implications of trying multiple tests for an analysis

Let's say I have a fixed set of data containing one variable. I'm interested in whether the variable shows a difference between two conditions. Now I perform a test and determine a p-value somewhere above my desired alpha. Being disappointed, I iterate over multiple different alternatives of tests for the same hypothesis and find that one test gives a p-value below my alpha.

This seems to be some kind of "fishing expedition", but not of the kind that a Bonferroni correction is supposed to fix: I'm always testing the same hypothesis. Still, somehow my gut feeling tells me this is fishy. What is generally considered appropriate here? Do the fishing and report it in the paper? Go with the first p-value? Calculate the mean p?

• You are right that this is widely regarded as dubious, but how dubious? You would not fault yourself for trying different hammers or screwdrivers before deciding on the best for a job. How far is this similar? On the strictest interpretation, one should specify the tests to used in advance and use only those tests. That's admirably puritan, but I'd guess it's minority practice. In what I read, I often think about a test that has been used as just another kind of exploratory data analysis, especially when I would tend to favour a different test or no test or all (e.g. a graphical approach). Commented Jul 5, 2013 at 11:48

If the different tests really test the same null hypothesis and result in different $p$-values, that it is an issue of power. Typically, there is a literature comparing these tests, and you can just look up which test best applies to your situation.
There's no need to assume that different null hypotheses are being tested to use the Bonferroni correction. (See Cox & Hinkley (1974), Theoretical Statistics, 3.4 (iii) for a derivation & discussion of its application to multiple tests of the same null hypothesis—your question involves optional stopping rather than choosing the lowest p-value for a fixed no. tests, but I'm assuming that wasn't the important point.) If you really were going to keep on performing arbitrary tests until you get a p-value you like, a multiple-testing correction would seem more relevant than it usually does. Bear in mind that any sample is extreme in some respect, so you can always find a test that will reject the null hypothesis at an $\alpha$ of your choosing. In practice using a test statistic that's obviously post hoc or outré will excite suspicion (see Witztum et al. (1994), Equidistant Letter Sequences in the Book of Genesis, Statistical Science, 9, 3).