Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite

I have a sort of philosophical question about when multiple comparison correction is necessary.

I am measuring a continuous time varying signal (at discrete time points). Seperate events take place from time to time and I would like to establish if these events have a significant effect on the measured signal.

So I can take the mean signal that follows an event, and usually I can see some effect there with a certain peak. If I choose the time of that peak and do say a t-test to determine if it is significant vs when the event doesn't occur do I need to do multiple comparison correction?

Although I only ever performed one t-test (calculated 1 value), in my initial visual inspection I selected for the one with the largest potential effect from the (say) 15 different post delay time points I plotted. So do I need to do multiple comparison correction for those 15 tests I never performed?

If I didn't use visual inspection, but just did the test at each event lag and choose the highest one, I surely would need to correct. I am just a little confused as to whether I do need to or not if the 'best delay' selection is made by some other criterion than the test itself (eg visual selection, highest mean etc.)

share|improve this question

5 Answers

up vote 8 down vote accepted

Technically, when you do a visual preselection of where to do the test, you should already correct for that: your eyes and brain already bypass some uncertainties in the data, that you don't account for if you simply do the test at that point.

Imagine that your 'peak' is really a plateau, and you hand pick the 'peak' difference, then run a test on that and it turns out barely significant. If you were to run the test slightly more to the left or to the right, the result could change. In this way, you have to account for the process of preselection: you don't have quite the certainty that you state! You are using the data to do the selection, so you are effectively using the same information twice.

Of course, in practice, it is very hard to account for something like a handpicking process, but that doesn't mean you shouldn't (or at least take/state the resulting confidence intervals / test results with a grain of salt).

Conclusion: you should always correct for multiple comparisons if you do multiple comparisons, regardless of how you selected those comparisons. If they weren't picked before seeing the data, you should correct for that in addition.

Note: an alternative to correcting for manual preselection (e.g. when it is practically impossible) is probably to state your results so that they obviously contain reference to the manual selection. But that is not 'reproducible research', I guess.

share|improve this answer
1  
Always doing corrections though, inflates your type II error rate. If you have all significant results before correction, you may lose them all after correction, not accounting for the low odds of getting all significant results. This may depend on the cost of a type I or type II error in your context. – Etienne Low-Décarie May 5 '12 at 13:08
Nick gave the answer I would like to have given if I were first to respond. However in the initial setup you (mkpitas) said that if you actually performed the 15 tests you wouldn't have to do the multiplicity correction. I don't see why you would say that. I think in that case the need for multiplicity correction just becomes more obvious. @etienne your point applies to FWER correction which is very strict in controlling type I error. If you use FDR you won't sacrifice as much power. – Michael Chernick May 5 '12 at 13:36
up vote 6 down vote

Long ago, in one of my first statistics classes, I was reading about this in a text (I think it was an old edition of Cohen's book on regreession) where it said "this is a question about which reasonable people can differ".

It is not clear to me that anyone ever needs to correct for multiple comparisons, nor, if they do, over what period or set of comparisons they should correct. Each article? Each regression or ANOVA? Everything they publish on a subject? What about what OTHER people publish?

As you write in your first line, it's philosophical.

share|improve this answer
4  
You're right that there's a question about how many comparisons are being done, but I don't think it implies your conclusion. Reasonable people can differ because they have different objectives and different valuations (loss functions) for the possible outcomes. If you should be correcting for multiple comparisons, this is because it leads to better expected loss. As such, this is an intensely practical issue, not mere "philosophy," and there are rational ways to resolve it on which reasonable people can agree. – whuber♦ Oct 10 '11 at 14:37
1  
@whuber you are surely right in some situations. Sometimes there is a sensible loss function, although it is often hard to get one stated explicitly. But other times, e.g. in exploratory work, I have trouble seeing how any loss function is possible. Of course, the whole loss function idea gets us away from the grail-like stature of p = .05, and the typical assumption that power = .8 or .9 is good enough, and onto (to my mind) more sensible idea that we establish these on more substantive grounds. – Peter Flom Oct 10 '11 at 16:11
1  
Thank you for clarifying the scope and the spirit of your reply, Peter. – whuber♦ Oct 10 '11 at 16:51
3  
I get infuriated when people say multiplicity testing doesn't matter. I see this attitude expressed all too often in medical research. You can point to many papers that reached incorrect conclusions because multiplicity was ignored. It is critical to not publish papers with wrong conclusions in medicine because it affects how patients are treated and lives are at stake. Multiplicity contributes to publication bias (because when an issue is studied many times only the studies with significant results get published) which is a serious issue in meta analysis, – Michael Chernick May 5 '12 at 13:45
The FDA is acutely aware that if they don't make sure that type I errors are controlled in clinical studies they risk approving an unexceptable number of ineffective or unsafe drugs. This ahs been paricularly an issue for them when reviewing proposed adaptive designs. Their first concern is to make sure the design does not allow inflation of the type I error. – Michael Chernick May 5 '12 at 13:51
show 5 more comments
up vote 4 down vote

If you are trying to make one-off decisions about reality and want to control the rate at which you falsely reject the null hypothesis, then you will be using null hypothesis significance testing (NHST) and will want to use correction for multiple comparisons. However, as Peter Flom notes in his answer, it's unclear how to define the set of comparisons over which to apply the correction. The easiest choice is the set of comparisons applied to a given data set, and this is the most common approach.

However, science is arguably best conceived as cumulative system where one-off decisions are not necessary and in fact serve only to reduce the efficiency of evidence accumulation (reducing obtained evidence to a single bit of information). Thus, if one follows a properly scientific approach to statistical analysis, eschewing NHST for tools like likelihood ratios (possibly Bayesian approaches too), then the "problem" of multiple comparisons disappears.

share|improve this answer
up vote 1 down vote

A possible alternative to correction, depending on you question, is to test for significance of the sum of p-values. You can then even penalize yourself for test that are not done by adding high p-values.

Extension's (which don't require independence) of Fisher's method (which require independence of test) could be used.

Eg. Kost's method

share|improve this answer
These are examples of procedures used in meta-analysis when the individual studies only provide p-values or the data cannot be combined but each study has a p-value calculated. Also Fisher's combination method and inverse normal are ways to construct stopping rules in adaptive designs. – Michael Chernick May 5 '12 at 13:59
up vote 1 down vote

One very important thing to remember is that multiple testing correction assumes independent tests. If the data your analyzing isn't independent, things get a little more complicated than simply correcting for the number of tests performed, you have to account for the correlation between the data being analyzed or your correction will probably be way too conservative and you will have a high type II error rate. I've found cross-validation, permutation tests, or bootstrapping can be effective ways to deal with multiple comparisons if used properly. Others have mentioned using FDR, but this can give incorrect results if there's a lot of non-independence in your data as it assumes p-values are uniform across all tests under the null. The distribution of p-values across tests under the null can be very skewed if a lot of non-independence exists.

share|improve this answer
Welcome to the site, Matt. Regarding your opening sentence: One very important thing to remember is that multiple testing correction assumes independent tests. Note that this is true for some multiple-testing correction procedures, but certainly not all. For example, the simplest of all (Bonferroni) makes no independence assumption, and, indeed is quite inefficient if the tests actually are independent! :-) Also, in a continuous-distribution setting, the (marginal) distribution of a single $p$-value will be uniform under the null; you might consider editing to clarify your remarks. – cardinal Jun 28 '12 at 16:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.