I have a sort of philosophical question about when multiple comparison correction is necessary.
I am measuring a continuous time varying signal (at discrete time points). Seperate events take place from time to time and I would like to establish if these events have a significant effect on the measured signal.
So I can take the mean signal that follows an event, and usually I can see some effect there with a certain peak. If I choose the time of that peak and do say a t-test to determine if it is significant vs when the event doesn't occur do I need to do multiple comparison correction?
Although I only ever performed one t-test (calculated 1 value), in my initial visual inspection I selected for the one with the largest potential effect from the (say) 15 different post delay time points I plotted. So do I need to do multiple comparison correction for those 15 tests I never performed?
If I didn't use visual inspection, but just did the test at each event lag and choose the highest one, I surely would need to correct. I am just a little confused as to whether I do need to or not if the 'best delay' selection is made by some other criterion than the test itself (eg visual selection, highest mean etc.)