4
$\begingroup$

I noticed that most RCTs use simple univariate statistical test to perform sample size calculation. For example for continous outcome, z-test may be used. But eventually for final analysis, mixed model for repeated measures adjusting for randomization stratification factors may be used to perform the analysis. This is usually the case when the outcome is measured at multiple timepoints, although the primary endpoint is at one timepoint only.

Would the sample size calculation remain valid here? Wouldn't the sample size estimate be not accurate?

Edit: Could the sample size calculated be acceptable since multivariate and repeated measure methods almost always have higher power than univariate methods?

$\endgroup$
1
  • 1
    $\begingroup$ There's a common assumption, which I think should be correct for any sensibly-designed RCT, that yes, a simple test will have lower power than a more appropriate complicated model, so the reported power calculation is a lower bound. $\endgroup$
    – Eoin
    Commented Feb 29 at 9:55

1 Answer 1

2
$\begingroup$

If your impression is correct - I won't pretend to enough knowledge of the field to confirm or dispute it - then yes, the sample size calculation would be incorrect, possibly wildly so. A sample size calculated under the assumption of some effect size $d$ and some power $\beta$ can be much larger or much smaller than the sample size required to detect a different effect size (typically measured in some other way, say $\eta^2$ rather than $d$ in an ANOVA, or some version of $R^2$) in a more complex model.

As to why this is so, it's probably a case of the drunk person searching for their car keys under the lamp post, not because they lost the keys there, but because the light is better there. Or, more explicitly and charitably, it's easier to perform a power analysis for a simple (say) t-test. There are tools and even online calculators where you only need to input an effect size and a power and make a few decisions as to what kind of test you have in mind, and the required sample size drops out.

In contrast, determining the sample size for a more complicated model ("mixed model for repeated measures adjusting for randomization stratification factors") is... more complicated. You need to make lots of assumptions as to correlation structures between your repeated measures, and as to the effects of your stratification, probably about both the distributions and the effects of any covariates. Once you have made your assumptions, the only way to actually determine a sample size is likely to simulate your data and your analysis many times, check whether power is appropriate, and adjust the sample size up or down until you reach the specified power. And then you really need to do this over again multiple times to get an understanding of the sensitivity of your results to your assumptions. This is much more complex, and requires thinking deeply about your analysis before planning your study - which you should be doing anyway, but which non-statisticians are understandably not very interested in (we have many questions here on CV where the correct answer is "you should have thought about this before data collection").

Except for that last point about thinking about this before the study, I find it hard to fault clinical researchers about this shortcut. They typically don't have the funds to pay for a statistical expert, and need to muddle through on their own. And if they have a hard page limit of (say) 12 pages for their grant proposal, you can't very well expect them to spend 2 pages on the assumptions and the specific model they used for their sample size calculation, plus the results of a sensitivity analysis.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.