How to calculate sample size and prove non-inferiority/equivalence between two risk prediction models I am in need of determining the appropriate sample size to allocate to a test set in order to prove equivalence/non-inferiority of two prediction models.
My model is a deep learning based model (using DeepHit) to predict risk of sustaining at least 1 fracture utilizing the actual outcome data for a population. I want to prove it is non-inferior to an existing method (FRAX). Both my model and FRAX can produce a 10 year probability of fracture which can be compared to the gold-standard (i.e. did the event actually occur or not). I can do c-index and brier scores to compare model performance.
Clinically, patients with > 20% 10 year risk of fracture are considered for treatment (which can be thought of as a binary classification). I believe I will need to compare diagnostic performance and AUC. The problem with this dichotomy is that any model will have low sensitivity since by definition patients at high risk will still only experience the event > 20% of the time.
So my question is how do I determine the appropriate sample size to sufficiently power the results from my test set? I plan to stratify the test set by male/female and above/below age 65 years of age based on event prevalence.
 A: I see two statistical problems here that can be worked around with a different approach.
The first is using a split sample for model building and evaluation. The risk with the split is that you lose both power in building the model and precision in testing the model. Frank Harrell estimates that you need at least 20,000 subjects to get around those combined risks. On that basis, you don't have enough cases to provide adequate power for the split-sample approach.
The second is how to apply the principles of non-inferiority testing here. That's usually done with an outcome model, to evaluate non-inferiority of some treatment versus a standard of care. The treatment effect is typically represented in a survival model by a coefficient that follows an asymptotic normal distribution, so that its standard error can readily be used to test for a pre-specified level of non-inferiority.
Here, the question is whether the new model type provides non-inferior predictions versus the well-established FRAX tool. If you were to evaluate with the Brier score (better than the C-index for model comparison),* I'm not sure that there's a theoretical form for the distribution of Brier scores in this context to allow for a corresponding parametric test.
Applying the optimism bootstrap should get around both problems. This uses all of your data to build the model while providing a reasonable evaluation of how your modeling approach would work when applied to a new sample from your underlying population, without depending on a parametric distribution for testing.
The idea is that bootstrap re-sampling from your data set mimics the process of taking your data set from the underlying population. Thus repeated model building on bootstrapped sample of your data and testing all of them on your full data set provides estimates of both bias and variance to be expected when building your type of model on multiple samples of the underlying population.
What's required is to repeat all of your modeling steps on each of the bootstrap samples: in your case, all the steps for setting up the DeepHit neural-network survival model. Then the distribution of Brier scores among the bootstrap-based models, all evaluated on the full data set, will show you how frequently a Brier score from your DeepHit modeling process will be worse than some pre-specified difference from the FRAX Brier score.
That said, you might be expected to deal with questions about whether this type of model will be better suited to clinical practice than the FRAX score, which is based on a few easy-to-understand risk factors and either a bone-density measurement or a body-mass index. Neural networks aren't noted for producing models that are easy for humans to grasp; see this discussion, for example, about the tradeoffs between machine-learning models and more traditional statistical models in practice. One of the big advantages of FRAX is how easy it is to explain to a patient as a basis for guiding treatment decisions. In contrast, even the model builder might be hard-put to explain why a neural network decided that a particular patient is thought to have a high risk.

*See the discussion here for example. The C-index only addresses discrimination in terms of rank-order of probabilities (maybe not very useful when most cases don't have events), while the Brier score helps evaluate calibration.
