I am running a number of experiments where I obtain a large number of observations with one variable (yielding two sets of values; many more than would be needed for a hypothesis test). I want to test whether the values from the one set are significantly higher/lower than the other set. As the values seem to follow a normal distribution, I assume that a two-sample t-test should work fine?

I also want to see how different pairs of sets compare to each other. E.g. is the difference between setx and sety "easier" to observe than the difference between seta and setb, assuming that for both tests the t-test finds a significant difference. I was looking into power analysis, but this seems to mainly target a priori calculations.

Finally, because under certain conditions, it can be difficult to obtain as many observations, I also wanted to know what the minimum number of observations per set that is required to achieve a certain level of significance/effect. (note: under other conditions, observations follow the same distribution as the "superset" I collected) It seems that power analysis is also a viable approach for this, but I can't quite figure out how to apply it a posteriori.

Any tips or pointers would be very helpful (I'm very new to statistics!)


  • $\begingroup$ I think you should not pursue the power analysis approach. This seems like a convoluted metric to use for distinguishing populations. Is there a good reason not to use a confidence interval for the difference in means? $\endgroup$ – HStamper Mar 8 at 1:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.