Are multiple comparisons corrections necessary for informal/visual "multiple comparisons"? I have a sort of philosophical question about when multiple comparison correction is necessary.
I am measuring a continuous time varying signal (at discrete time points). Separate events take place from time to time and I would like to establish if these events have a significant effect on the measured signal.
So I can take the mean signal that follows an event, and usually I can see some effect there with a certain peak. If I choose the time of that peak and do say a t-test to determine if it is significant vs when the event doesn't occur do I need to do multiple comparison correction?
Although I only ever performed one t-test (calculated 1 value), in my initial visual inspection I selected for the one with the largest potential effect from the (say) 15 different post delay time points I plotted. So do I need to do multiple comparison correction for those 15 tests I never performed?
If I didn't use visual inspection, but just did the test at each event lag and choose the highest one, I surely would need to correct. I am just a little confused as to whether I do need to or not if the 'best delay' selection is made by some other criterion than the test itself (e.g. visual selection, highest mean etc.)
 A: Long ago, in one of my first statistics classes, I was reading about this in a text (I think it was an old edition of Cohen's book on regreession) where it said "this is a question about which reasonable people can differ".
It is not clear to me that anyone ever needs to correct for multiple comparisons, nor, if they do, over what period or set of comparisons they should correct. Each article? Each regression or ANOVA? Everything they publish on a subject? What about what OTHER people publish? 
As you write in your first line, it's philosophical. 
A: If you are trying to make one-off decisions about reality and want to control the rate at which you falsely reject the null hypothesis, then you will be using null hypothesis significance testing (NHST) and will want to use correction for multiple comparisons. However, as Peter Flom notes in his answer, it's unclear how to define the set of comparisons over which to apply the correction. The easiest choice is the set of comparisons applied to a given data set, and this is the most common approach.
However, science is arguably best conceived as cumulative system where one-off decisions are not necessary and in fact serve only to reduce the efficiency of evidence accumulation (reducing obtained evidence to a single bit of information). Thus, if one follows a properly scientific approach to statistical analysis, eschewing NHST for tools like likelihood ratios (possibly Bayesian approaches too), then the "problem" of multiple comparisons disappears.
A: One very important thing to remember is that multiple testing correction assumes independent tests. If the data your analyzing isn't independent, things get a little more complicated than simply correcting for the number of tests performed, you have to account for the correlation between the data being analyzed or your correction will probably be way too conservative and you will have a high type II error rate. I've found cross-validation, permutation tests, or bootstrapping can be effective ways to deal with multiple comparisons if used properly. Others have mentioned using FDR, but this can give incorrect results if there's a lot of non-independence in your data as it assumes p-values are uniform across all tests under the null. The distribution of p-values across tests under the null can be very skewed if a lot of non-independence exists.
A: Technically, when you do a visual preselection of where to do the test, you should already correct for that: your eyes and brain already bypass some uncertainties in the data, that you don't account for if you simply do the test at that point.
Imagine that your 'peak' is really a plateau, and you hand pick the 'peak' difference, then run a test on that and it turns out barely significant. If you were to run the test slightly more to the left or to the right, the result could change. In this way, you have to account for the process of preselection: you don't have quite the certainty that you state! You are using the data to do the selection, so you are effectively using the same information twice.
Of course, in practice, it is very hard to account for something like a handpicking process, but that doesn't mean you shouldn't (or at least take/state the resulting confidence intervals / test results with a grain of salt).
Conclusion: you should always correct for multiple comparisons if you do multiple comparisons, regardless of how you selected those comparisons. If they weren't picked before seeing the data, you should correct for that in addition.
Note: an alternative to correcting for manual preselection (e.g. when it is practically impossible) is probably to state your results so that they obviously contain reference to the manual selection. But that is not 'reproducible research', I guess.
A: A possible alternative to correction, depending on you question, is to test for significance of the sum of p-values.  You can then even penalize yourself for test that are not done by adding high p-values.
Extension's (which don't require independence) of Fisher's method (which require independence of test) could be used.
Eg. Kost's method
