I've got a study, a before-and-after controlled design.

I have pre-intervention data and post-intervention data, the intervention is an educational intervention, given to general practitioners and the outcomes (a continuous outcome and a binary outcome) are measured on patients. Therefore, the pre-intervention group is different to the post-intervention group. I also have a control group, collected over the same time period. So I have n1 pre-intervention control, n2 post-intervention control, n3 pre-intervention active and n4 post-intervention active patients.

I'd be grateful on your thoughts on how to evaluate the intervention. The groups (n1, n2, n3 and n4) are small, roughly 25-30. Thus I am also assuming there is no underlying trend in the change in the outcome prior or post intervention (I have insufficient numbers for an interrupted time series approach or segmented regression).

Clearly calculating the difference in post-intervention data ignores any potential differences in pre-intervention levels. Therefore, one approach could be to calculate 83.4% confidence intervals in the pre-post difference in both the control and active arms and observe if the two 83.4% CIs overlap to draw conclusions at the 95% level.

Any suggestions on alternative approaches?

  • 2
    $\begingroup$ Why would you consider 83.4% CIs? $\endgroup$ – chl Jul 1 '13 at 19:58
  • $\begingroup$ How do you intend to compute a pre-post difference if the pre and post-test measures are not matched? $\endgroup$ – Gala Jul 1 '13 at 20:21
  • $\begingroup$ @chl To implicitly have a test of the difference being 0 at the 5% level. This is completely unrelated to the issue of what to do with pre-intervention measures when the patients are not matched. $\endgroup$ – Gala Jul 1 '13 at 20:24
  • $\begingroup$ @Gael Lauransto get a pre-post difference, it boils down to a 2-sample problem. So I can calculate the mean pre intervention score, the mean post intervention score etc, get a standard error and calculate the confidence interval. $\endgroup$ – Pippa West Jul 1 '13 at 20:28
  • $\begingroup$ OK, I see. Another question: Do you have one patient per GP? And were the GP randomly assigned to treatment or control? $\endgroup$ – Gala Jul 1 '13 at 20:53

Comparing overlapping 95% intervals is problematic, the nominal level to test for a difference is no longer 5%. There is a large body of literature on this. If two CIs don't overlap, there isn't a problem, however, if they (95%CIs) do overlap, they can still be different.

If a difference is what is of interest, then calculating a CI around the difference should be done, and not comparing two 95% CIs.

  • $\begingroup$ Hence the 83.4% CI as mentioned in the second reference (that's assuming independence and equal variance, of course). $\endgroup$ – Gala Jul 1 '13 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.