1
$\begingroup$

I have not done a power analysis in a long time, but was asked to do so for a new study. The design is as follows:

  • At enrollment, every patient has no diagnosis because conventional testing did not detect any disease
  • The patients are then given a new test
  • The results of the new test is given to the clinician, who then makes changes to treatment as needed.
  • The clinician would then determine whether the test result had a postivie impact on clinical care for the patient (Yes or No)

The goal is to find out what is the sample size needed to detect a positive impact in 10%, 20%, 30%, or 40% of the patients. I'm a little confused as to how to do a power analysis based on percentage impact. Any guidance would be well appreciated!

Improve this question
$\endgroup$
5
  • $\begingroup$ If the difficulty is that you are used to using differences to compare outcomes, then--at least conceptually--express all outcomes as logarithms. The percent differences translate to numerical differences in the logarithms, so nothing is new here and you can proceed as always. Is there something that might stand in the way of such an approach? $\endgroup$
    – whuber ♦
    Commented Sep 2, 2022 at 19:14
  • $\begingroup$ @whuber I suppose that could work. Would the approach then be to do a power analysis for a paired sample t-test? $\endgroup$
    – st94070
    Commented Sep 2, 2022 at 19:45
  • 1
    $\begingroup$ I'm a bit worried by the outcome measure: "The clinician would then determine whether the test result had a positive impact on clinical care for the patient." There would seem to be a big risk of what's sometimes called the "Hawthorne Effect." Even putting that aside, will each physician assign a "percentage impact" for each case, or is this a yes/no "impact" assessment that will be averaged over cases/physicians in some way to get a percentage? The way to proceed could differ, depending. Please edit the question to provide that information. $\endgroup$
    – EdM
    Commented Sep 2, 2022 at 21:11
  • $\begingroup$ @EdM I clarified the assessment in the question. I'm not too concerned about the Hawthorn effect given the clinicians are already doing a thorough assessment for presence of disease given that is an entry requirement. $\endgroup$
    – st94070
    Commented Sep 2, 2022 at 22:13
  • $\begingroup$ I think it would be very helpful to find out what statistical test or tests you are going to run on this problem before trying to figure out how to do a power analysis. This seems to be two separate issues lumped into one. $\endgroup$
    – Shawn Hemelstrand
    Commented Sep 2, 2022 at 23:54

1 Answer 1

Reset to default
1
$\begingroup$

Without a control group and without a way for a clinician to say that the test had a "negative impact" on clinical care, I'm a bit torn about providing an answer. With the current design, you are necessarily biased toward finding some "positive impact." The following will also provide some guidance if the design is improved.

Without a control group, your outcome will be the fraction of patients in which the clinician decided that the new test had "positive impact." That's just a binomial proportion, although your study design requires some further consideration described below.

Start with the (unrealistic) assumption that the "positive impact" determinations are independent of each other. You calculate the fraction of "positive impact" cases at the end of the study. Whether that ends up being 10% or 20% or 40%, that's a point estimate of the fraction regardless of sample size.

The sample size has to do with how closely you want to estimate that fraction, typically evaluated with the confidence interval (CI) around the point estimate. There are several ways to calculate CI for binomial proportions. The binconf() function in the R Hmisc package provides 3 of them. For example, if you have a 20% "positive impact" among 100 patients, your 95% CI (the default) would be estimated as:

Hmisc::binconf(20,100,method="all")
#            PointEst     Lower     Upper
# Exact           0.2 0.1266556 0.2918427
# Wilson          0.2 0.1333669 0.2888292
# Asymptotic      0.2 0.1216014 0.2783986

Are those limits narrow enough? If not, see what happens with more cases at the same estimated "positive impact":

Hmisc::binconf(100,500,method="all")
#            PointEst     Lower     Upper
# Exact           0.2 0.1658001 0.2377918
# Wilson          0.2 0.1672855 0.2372891
# Asymptotic      0.2 0.1649391 0.2350609

Proceed similarly for any combination of sample size and assumed "positive impact."

The assumption of independent assessments probably isn't wise here. Clinicians are likely to differ in their tendency to call a "positive impact"; your model needs to take that into account in some way. If you have 6 or more clinicians you could extend this to a binomial (logistic) regression with the clinicians as a random (intercept) effect. That can be implemented, for example, via the lmer() function in the R lme4 package

In that situation you would be wise to simulate large data sets based on assumptions about the "positive impact" overall and about how much clinicians will differ in terms of assigning a "positive impact." Then sample from those data sets repeatedly (allowing for differences among clinicians in the numbers of cases) and perform your binomial regression on each sample to get an idea of how narrow your confidence intervals will be.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for this detailed answer! I think your argument for the flawed design, which I was having trouble wrapping my head around, hits the nail on the head. I'll go back to the study lead to think about the design some more. $\endgroup$
    – st94070
    Commented Sep 6, 2022 at 6:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.