1
$\begingroup$

I’m running an initial study to decide whether a RCT is worth the time and resources. I have survey results from a treatment group, but unfortunately no control. Is there a reasonable way for me to compare my survey results to those in the literature (which do not have the treatment, but use an identical survey instrument)?

More specifically, could I use a one-sample t-test, where my survey results are the sample and I estimate the population mean from what I find in the literature? Or a two-sample test where the results from the literature are the second sample?

$\endgroup$

1 Answer 1

0
$\begingroup$

The sources of outcome variability here include: that among individuals having the same treatment or control, that between treatment and control, that between control groups in different populations, and possible interactions between treatment and populations. What you propose is risky at best, as there could be systematic differences between your population and the populations evaluated in the studies from the literature.

A one-sample t-test evaluated against an overall fixed value would be particularly risky. It wouldn't take the variability in outcomes among populations in the published studies into account at all.

Even a simple 2-sample t-test as you seem to propose could be misleading. That would use the standard error of outcomes among individuals in your population and the standard error of outcomes among populations in literature studies for the comparison. Those aren't really the same thing.

What you might do is to evaluate whether your treatment-group outcomes are far beyond the distribution of control-group outcomes in the literature, based on the standard deviations among the populations. If so, you might be confident that your values represent a treatment effect instead of just a population with a somewhat extreme control-group value and no true treatment effect. The risk is that might be too stringent a requirement and you might not perform a study that could have been useful.

It would be best to get at least a pilot sample of control outcomes for your population.

$\endgroup$
2
  • $\begingroup$ Thank so you much! I'll push a little harder for a sample of control outcomes. Could you elaborate on what you said in your last paragraph, "evaluate whether your treatment-group outcomes are far beyond the distribution of control-group outcomes in the literature, based on the standard deviations among the populations"? Are you just suggesting that I use standard deviation instead of standard error, or is there more to that? $\endgroup$
    – Rob L
    Commented Jul 10, 2023 at 22:56
  • $\begingroup$ @RobL you should use your understanding of the subject matter to determine just how far your treatment-group outcomes are from what might be a reasonable estimate of what control-group outcomes would be in your population. The standard deviation of control outcomes among other populations provides some information about that, but you might be able to find other useful information. Best is to get an estimate of control-group outcomes for your own population. $\endgroup$
    – EdM
    Commented Jul 11, 2023 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.