# 1-sample vs 2-sample test question

I have a situation where I measure something for a long time, and occasionally a seperate event of interest will occur. I want to determine if these events have a significant effect on the signal I am measuring. So I am thinking of taking the signal value at each event time, and comparing it to times when there was no event.

But as I see it there are a couple of ways to do this. First, I could take the whole population mean over all the data I have collected (ignoring presence or absence of an event), then do a 1-sample t-test on the event present data set against the whole data mean. Or I could take the mean over all the data, but excluding the times the event happens but still do a 1-sample test against the measured no-event mean.

Or I could take my set of event responses and create a surrogate 'no-event' data set of the same size from points chosen at random when there was no event and do a 2-sample t-test between these two. This surrogate set could have the same number of points as the event set, or could have many more (I have much more no-event data than event data).

So I am in a bit of a muddle as to which of these would be the best in terms of statistical power and avoiding any mistakes. Any advice is appreciated. I would especially be interested in understanding conceptually the reason for any preference of one method over another.

-

Any of the methods that you mentioned would work. I would avoid the 2-sample where you compare to a subset of those values where the event did not occur, using less data will cost you power and not give you any advantages. I would probably go with the 2-sample t-test where the 2nd group is all the observations that don't correspond to the event.

Another test that you did not state above is a permutation test. You can simply take samples from the entire dataset equal in size to your number of events and compute the mean, do this a bunch of times, then compare that distribution of means to the mean from your sample corresponding to the event. The p-value is the number of means that are equal to or more extreeme than the actual sample mean. This is nice in that it requires fewer assumptions (don't need normality or near normality and other assumptions) and could also be used with other statistics (median, variance, etc.) if those are of more interest to you.

-
Thanks, for the permutation approach I take samples from the entire dataset at random, without consideration of event / no-event (ie it would be wrong to do the permutation test just on no-event data)? –  thrope Oct 12 '11 at 9:31
The null hypothesis is that the event related measurements are the same as the non-event ones, so you should sample from all of them. The observed group should be a possibility in the samples you take. –  Greg Snow Oct 12 '11 at 16:59