Implications of trying multiple tests for an analysis

Question

Let's say I have a fixed set of data containing one variable. I'm interested in whether the variable shows a difference between two conditions. Now I perform a test and determine a p-value somewhere above my desired alpha. Being disappointed, I iterate over multiple different alternatives of tests for the same hypothesis and find that one test gives a p-value below my alpha.

This seems to be some kind of "fishing expedition", but not of the kind that a Bonferroni correction is supposed to fix: I'm always testing the same hypothesis. Still, somehow my gut feeling tells me this is fishy. What is generally considered appropriate here? Do the fishing and report it in the paper? Go with the first p-value? Calculate the mean p?

Answer 1

Often these different tests test subtly different hypotheses. For example, many tests that are used to check for normality/Gaussianity of a distribution actually look at very specific deviations from normality (e.g. no skewness) and tests differ with respect to what quantity they exactly look at. So I would start with finding out what the difference is between these tests and decide which hypothesis is the one that is really of interest to you.

If the different tests really test the same null hypothesis and result in different $p$-values, that it is an issue of power. Typically, there is a literature comparing these tests, and you can just look up which test best applies to your situation.

Having said that, my first guess would be that your different tests actually test similar but not the same hypotheses. In general, once you understand where the differences between tests come from, the choice becomes pretty obvious.

Answer 2

There's no need to assume that different null hypotheses are being tested to use the Bonferroni correction. (See Cox & Hinkley (1974), Theoretical Statistics, 3.4 (iii) for a derivation & discussion of its application to multiple tests of the same null hypothesis—your question involves optional stopping rather than choosing the lowest p-value for a fixed no. tests, but I'm assuming that wasn't the important point.) If you really were going to keep on performing arbitrary tests until you get a p-value you like, a multiple-testing correction would seem more relevant than it usually does. Bear in mind that any sample is extreme in some respect, so you can always find a test that will reject the null hypothesis at an $\alpha$ of your choosing. In practice using a test statistic that's obviously post hoc or outré will excite suspicion (see Witztum et al. (1994), Equidistant Letter Sequences in the Book of Genesis, Statistical Science, 9, 3).

As to what to report: there is a distinction between exploratory & confirmatory analyses. Real analyses often have elements of both. Best to simply tell the truth about what you did, & let your readers decide for themselves whether they're bothered or not by multiple testing.