# Sample size and a study design: AI in shadow mode vs physicians

My research group has a group of patients where we are comparing the performance of a Random Forest classifier for acute respiratory tract infections. We are designing a study where we compare the output of the classifier to a physician. The patients first answer questions before meeting with the physicians and we get the rest of the data needed from the text note from the consultation. The outcomes are the rate of antibiotic prescriptions and chest x ray orders (and if they are positive vs negative).

My questions are on sample size and on which kind of study would fit best. With an alpha of 0.05 and a power of 0.8, how would one calculate a sample size? I am guessing a type of observational study design would fit best to be able to reject the null hypotheses presented regarding the performance of the classifier vs the physicians.

I already found a reference for how to calculate the sample size if we were to observe two groups with different risk factors and measuring the outcome (http://njppp.com/fulltext/28-1567942207.pdf) but wouldn't it make more sense to have a single group and have the classifier work in "shadow mode"?

EDIT: We have done a retrospective analysis which shows that antibiotic prescriptions are prescribed in 25% of cases which can be reduced to mean 18% and the chest x-rays are at 12% and can be reduced to mean 9% (without missing a positive x-ray).

EDIT: The ground truth/gold standard for the antibiotics will be a panel of 5 physicians which will go through each case in the validation set. They will estimate if there was a need for antibiotics or not (by a majority vote). With each patient we have a binary value indicating if an antibiotic was prescribed or not. The ground truth for the chest x rays are if the x-ray is positive for pneumonia specific changes as per description of the radiologist.

EDIT: The study concludes by comparing the rates of chest x-rays orders and antibiotic prescriptions recommended by the classifier vs the physician who originally saw the patient. These are binary values. We also look at the results of the chest x-rays to determine if the image was positive for pneumonia or not which is also a binary value.

• Please say more about the "ground truth" in this situation and how you estimate it. That's what you're using to evaluate both the random forest and the clinician results. It's obviously desirable to minimize unnecessary antibiotics or imaging, but there's a tradeoff in making false-negative decisions, also. So the nature and quality of the "ground truth" values is critical here. Please provide that information by editing the question, as comments are easy to overlook and can be deleted.
– EdM
Sep 25, 2021 at 15:26
• Thanks for the feedback, I have edited the question. Sep 25, 2021 at 15:58
• Can you give concrete examples of your outcomes? What would the data look like at the end of the study? Sep 25, 2021 at 16:24

Power analysis to determine sample size depends on having estimates of the variability in what you're trying to measure and the magnitude of any differences that you're trying to find (often called "effect size"). Generic methods like those in your linked resource or implemented in software like G*Power take such estimates together with your desired significance levels and power to find the sample size for particular types of design. Yes, as you are evaluating paired comparisons instead of separate groups you would use corresponding paired instead of independent-group values for variability and effect size.

You seem, however, to have a fair amount of preliminary data on this. I thus suggest that you do your power calculation by simulating a large set of new values based on your preliminary data, trying your intended analysis on those simulated data, and seeing what sample size is needed based on your simulations of your particular situation.

That might seem daunting at first, but it's pretty straightforward. You already have information on test results and clinical annotations on many patients, the associated probabilities that clinicians versus the random forest (RF) would "prescribe" antibiotics or imaging, and the corresponding true findings of pneumonia. As those tend to be all-or-none decisions or results, each of those could be modeled with logistic regressions based on the data that you have. Next, you use those logistic-regression models to "predict" pneumonia and clinician versus RF-based recommendations for antibiotics and imaging for a very large number of "new patients" based on the distributions of clinical data among the patients in your population of interest. That gives you a set of simulated data. Then you perform your intended analyses on the simulated data, evaluating the sample size needed to detect the differences of interest.

This approach has a few advantages. First, instead of depending on crude generic estimates of variability and effect sizes you use estimates based on the data most directly related to your study: the data you already have. Second, it forces you to think carefully in advance about the statistical tests that you want to perform on your data. A crude power calculation with a generic tool might not end up working well for the specific test that you really want to perform. Third, through the simulation and analysis steps it forces you to think carefully about the nature of your data and the (potentially hidden) assumptions that you are making.

As you do this, however, I would recommend that you focus on probabilities instead of all-or-none classifications as much as possible. Although a clinician's decisions to prescribe antibiotics or order imaging are all-or-none decisions, those decisions are based on the clinician (perhaps unconsciously, or based on years of training and experience) putting together an estimate of the probability of a patient's having pneumonia with the costs and benefits of the drug or the imaging. Your RF presumably returns a probability of needing antibiotics or imaging; the probability cutoff that you choose for the RF implicitly represents those cost/benefit tradeoffs. Even your "ground truth" for antibiotics doesn't need to be all-or-none. If only 3 out of 5 clinicians think that antibiotics should have been prescribed in a case, maybe you should represent that as a 60% probability of needing antibiotics instead of a fixed "antibiotics-YES" decision.

Implementing the sampling from the data

As noted in a comment, you can't just sample from the marginal distributions of the patient characteristics separately, as they are highly correlated. One way to approach this problem would be to model those correlations directly and sample from the correlation model. A second would be to stick with the set of patients for which you already have clinical data but sample from their probability distributions of having pneumonia and needing antibiotics or imaging.

Modeling correlations among variables is often done with copulas. Copulas provide the link between the marginal distributions of individual predictors and their joint distributions. Recent work has shown how to extend methods originally focused on continuous joint distributions and parametric copula forms to mixed discrete and continuous distributions and to non-parametric copulas that might be needed for your data set.

A second approach, which might be simpler to implement, would be to extend your current data set to incorporate the variability in predictions from your model. For example, build a binomial model of pneumonia yes/no based on your current (pre-diagnosis) clinical data set. For each case in your data set, get the modeled probability of pneumonia. Make 100 copies of each patient's clinical data. Then label those copies as pneumonia yes/no in proportion to the modeled probability. For example, if the probability of pneumonia for a certain patient is 63%, label 63 of the patient's 100 data copies as pneumonia-yes and 37 as pneumonia-no.

That will give you an extended data set 100 times the scale of your current data set that incorporates your uncertainty in modeling pneumonia. Then you sample with replacement from that extended data set to evaluate the sample size needed for your study.

It sounds like you might be modeling all 3 outcomes of pneumonia, need for antibiotics, and need for imaging. If that's the case it might be best to model those all together in a multivariate binomial model that takes the inherent correlations among those outcomes into account. Then you would construct your extended data set in a way that incorporates those correlations among those outcomes. But the general principle would be the same.

The major point is that working with simulated data based on your current information is very fast, inexpensive, and flexible--particularly compared to what you would face if you rushed into an underpowered prospective trial based on a crude power estimate.

• Thank you for the feedback. I presume you are suggesting something similar as is described in the later part of this article? Sep 27, 2021 at 9:17
• @st0ne yes, that's the idea. The closer you base your simulation on the type of data and patient population that you have, the better your estimate is likely to be. For example, that web page seems to have sampled from a uniform distribution for age and equal male/female probability for its simulations. You want to sample from your actual distributions of ages, genders, clinical characteristics, pneumonia incidence, etc.
– EdM
Sep 27, 2021 at 11:57
• that makes sense. We would have to find a way to sample our data in a way so that the simulated data captures the correct probability distributions we are sampling our simulation data from. Using age as an example, if we are to simulate an 80 year old vs a 25 year old patient, the probability distribution for the same predictor would be quite different. I don't presume you have any good suggestions or links on how to execute this kind of sampling in python? Group by age and sex perhaps? Sep 27, 2021 at 12:53
• This approach is intriguing but a bit tricky to implement with the data we have collected. We tried sampling from age buckets (5 years apart) but the problem is that it's hard to sample in such a way that symptoms / clinical features correlate. E.g. if a patient has COPD, it is more likely he has low oxygen saturation. Our data has countless of these correlated examples. The logistic regression model therefore predicts that no images should be taken on the simulated patients. We would have to iterate over each feature and create a unique distribution for each. All suggestions appreciated Sep 28, 2021 at 10:45
• @st0ne you've already found one advantage of trying to do this: you found that you can't assume independence among the patient characteristics so you have to take the correlations among those characteristics into account. That also has implications for interpreting "predictor importance" and such in your models. I'll expand the answer in a day or so to suggest approaches based on copulas (to model the correlation structure) or, maybe more simply, working with your current set of patients but incorporating what you have already modeled with respect to probabilities of having pneumonia etc.
– EdM
Sep 28, 2021 at 14:23