I have a small time to event dataset (N=20) where patients are given one of two drugs (drug) at varying doses (dose). There are several biomarkers (biomarker1, biomarker2 etc) recorded for each patient included as covariates.

I'd like to estimate the hazard ratios of the biomarkers to determine if they are potentially prognostic of survival. As patients received different treatment regimens, I'm concerned about confounding and want to adjust for treatment when I consider the biomarkers in a Cox regression. If I had a large dataset, I would use a Cox model that adjusts for treatment and dose and includes all the biomarkers of interest

cph(Surv(time, event) ~ treatment + dose + treatment*dose + biomarker1 + biomarker2 ...)

I'm aware that in the development of prediction models, overfitting is a major concern as too many variables can cause overfitting and prevent the model from validating. As I'm interested in effects estimation to understand the data at hand, not building a prediction model, do I need to be concerned about overfitting and whether I should limit the number of included variables? If so, how should I determine how many covariates to include?

Do I need to limit the number of variables included in my model, or can I not worry about overfitting and use my full model with all treatment effects and biomarkers, understanding that the small N may cause my estimates to not replicate in future larger studies?


You did not state the all-important number of events, but clearly it cannot exceed 20.

There is a lower limit of sample size required to estimate even the simplest thing, such as an overall event incidence. If there were no censoring, the sample size required to estimate a simple proportion is 96. That is just to estimate the intercept in a binary logistic model. To have covariates you would clearly need more than 96 subjects. Any analysis of 20 subjects is futile.

A recent paper showing how to calculate sample size when covariates are present is here.

For information about where 96 comes from see BBR and RMS in these links.

  • $\begingroup$ Thanks for the answer Dr. Harrell. I suspected with 17 events it would be impossible to find anything. Is it safe to say then that when considering the number of covariates in a model, the sample size calculations from the paper you linked should be applied regardless of whether the model is intended for effect estimation vs prediction? $\endgroup$ – Tomas Bencomo Jul 14 '19 at 19:42
  • $\begingroup$ That's mainly right. I have another calculation in my RMS course notes that shows how many observations are needed in a Cox model just to estimate the underlying survival curves with no covariates. It's something like 150. So I don't think the dataset of 20 persons can be used for anything unless you had a single fully pre-specified risk factor to test for correlation with event time. $\endgroup$ – Frank Harrell Jul 15 '19 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.