4
$\begingroup$

Part of our samlpe has been collected (Phase 1, $n=12$ subjects with measurements pre and post treatments). The remaining sample (Phase 2 with another 8-10 subjects) will be collected in a few months.

Both phase 1 and phase 2 are supposed to be identical in terms of study design and treatment. The only reason for the second phase is that we were not able to obtain the desired number of subjects for the first phase. The final analysis of the treatment effect is suppose to include all subjects from phase 1 and phase 2.

While waiting for the rest of the data, is it ok to analyze the first part of the data? I feel hesitant to analyze the first part for the following reasons:

  1. The results of the interim analysis might affect the second phase. Depending on the results changes might be possible to the study design and/or treatment. However, an interim analysis (as in clinical trials) was never planned. A further concern is that the results might affect how/which patients are being allowed to participate in the second part. I doubt this would happen on purpose, but might be possible subconsciously and thus falsify the results.

  2. The results between the interim analysis and the final analysis might change. I might get a significant treatment effect for the phase 1 analysis which gets everybody excited. When analyzing the whole data the effect might vanish given the small sample. Now it's the statisticians fault (i.e. my fault) that the treatment effect is gone.

  3. Not enough power. Why even look at the data when just about half of the sample was collected?

Am I to overcautious or are this true concerns? What other problems might arise? Why and when would it be ok to look at the data prematurily? Would I have to adjust somehow the final analysis for having performed an interim analysis (assuming no changes in the design etc.)?

$\endgroup$
5
  • $\begingroup$ small sample is not a problem. If the experiment is well--designed and properly controlled. The statistics you may be calculating is based on certain methods. The methods are based on small sample theory as well as large sample theory. Choose a correct method to do analysis and you may validate the resula with whole data. Should be same even for whole data-set. $\endgroup$
    – user10619
    Commented Mar 20, 2015 at 16:20
  • 2
    $\begingroup$ Your concerns are well-founded. One solution is available by analyzing this as a group sequential design. Following such a protocol would rigorously justify the interim analysis, help produce correct final p-values, and perhaps even obviate any need for Phase 2. $\endgroup$
    – whuber
    Commented Mar 20, 2015 at 16:28
  • $\begingroup$ @whuber: Nice comment. Speaking about the group sequential design you've referenced, I'm curious about what is the statistical foundation/rationale for including the "failed" sample from previous phase into the next one. $\endgroup$ Commented Mar 20, 2015 at 16:45
  • 1
    $\begingroup$ @Aleksandr Although I'm not completely sure what you are asking, perhaps we ought to turn the question on its head: what would be the reason for ignoring data you have already collected? Not only does there seem to be no statistical justification for losing information, there would be both statistical and ethical reasons not to do something like this. If you drop data because they do not support your hypothesis, you would be demonstrably biasing the results. If you neither accounted for that bias nor documented what you have done, you would also be guilty of an ethical breach. $\endgroup$
    – whuber
    Commented Mar 20, 2015 at 16:58
  • $\begingroup$ @whuber: Thank you for your reply. It's clear to me now. Obviously, I didn't mean not to document dropping the initial data. But, I agree with you - dropping the data would introduce a serious bias (for some reason I forgot about this important statistical aspect - perhaps, it's time for my next cup of coffee). $\endgroup$ Commented Mar 20, 2015 at 17:10

1 Answer 1

1
$\begingroup$

I think that there is some confusion underlying this question. First of all, the answer very much depends what kind of analysis you're talking about - you're not clear on research design (despite mentioning of pre and post treatments) - observational study vs. experiment, true experiment vs quasi-experiment, with control group vs. without, whether you want/need to consider the effects of time (longitudinal design), whether you want/need to consider causal effects. For some exploratory data analysis (EDA) the Phase 1's sample size might not be a problem, but for some full-scale research designs even your combined sample might not be enough for valid statistical inferences.

Here are my thoughts on your particular concerns/reasons (I hope that this makes sense):

  1. You write: "the results of the interim analysis might affect the second phase". However, earlier you write: "the final analysis of the treatment effect is suppose to include all subjects from phase 1 and phase 2". It seems to me that there is a potential contradiction here, since if Phase 1 impacts Phase 2, how can you accurately account for that when doing final analysis, aggregating subjects from both phases? You might find helpful reading this chapter on treatment effects by Andrew Gelman.

  2. I agree with you on this concern. I would suggest collecting data from Phase 2 subjects, if you want to perform an aggregate analysis, that is use an aggregate sample.

  3. This is also a valid concern IMHO. Please see my points on the sample size in the beginning of this answer. However, nothing prevents you to assess the power of your potential analysis, using the Phase 1 sample size. But, even if it would be acceptable, the issues in #1 and #2 above might invalidate the results or conclusions anyway.

$\endgroup$
2
  • $\begingroup$ The answer meets the major concerns. Yet I am not sure why should an aggregate analysis be invoked. $\endgroup$
    – user10619
    Commented Mar 23, 2015 at 8:02
  • $\begingroup$ @subhashc.davar: Thank you! If the Phase 1 wouldn't reject the null hypothesis, then an aggregate analysis is recommended per group sequential design (see the link in the comments above). $\endgroup$ Commented Mar 23, 2015 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.