0
$\begingroup$

I'm a bit a new hypothesis testing, so I hope that this question makes sense. Say that I proposed a new treatment (like a new drug or something) and I wanted to test whether this drug was effective or not.

I understand that Power Analysis can be used to help us determine the sample size that is necessary to detect a significant effect. However, this analysis will not account for novelty effects of said treatment.

One thing that I never understood, where are novelty effects accounted for in power analysis? Hopefully, I can advice from an expert!

Thank you!

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ I suspect that what you mean by "novelty effects" has something to do with unpredicted deviations that may be due to the drug under investigation, and if so then the answer is that you have no way to predict the size of them or their variance so you cannot conduct a power analysis to determine sample size. (And note that you will need to do a confirmatory study before it can be safe to make an inference about them.) Is that what you had in mind? $\endgroup$ – Michael Lew Apr 2 '16 at 0:09
  • $\begingroup$ Yes, precisely. Thanks for letting me know. Say that I had some prior intuition on the "magnitude" of the novelty effect, is there a way I can choose a sample size to account for this novelty effect? $\endgroup$ – user46925 Apr 2 '16 at 14:28
  • 1
    $\begingroup$ The short answer is that you can do a power analysis if you are prepared to guess effect size and variance, but you should first think about whether you should be doing so. The longer answer is in my invited commentary on the American Statistical Association's recent statement on P-values amstat.tandfonline.com/doi/suppl/10.1080/00031305.2016.1154108 $\endgroup$ – Michael Lew Apr 3 '16 at 1:09
  • $\begingroup$ Wow cool! Thanks for the link to your article. It seems that novelty effects are very common - so I'm surprised I couldn't find anything after extensively Google searching. $\endgroup$ – user46925 Apr 3 '16 at 1:11
  • 1
    $\begingroup$ You should google "exploratory study" or search the distinction between primary and secondary outcomes rather than "novelty effects", as the latter is a phrase that I've never noticed before. $\endgroup$ – Michael Lew Apr 3 '16 at 3:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy