1
$\begingroup$

I have a small time to event dataset (N=20) where patients are given one of two drugs (drug) at varying doses (dose). There are several biomarkers (biomarker1, biomarker2 etc) recorded for each patient included as covariates.

I'd like to estimate the hazard ratios of the biomarkers to determine if they are potentially prognostic of survival. As patients received different treatment regimens, I'm concerned about confounding and want to adjust for treatment when I consider the biomarkers in a Cox regression. If I had a large dataset, I would use a Cox model that adjusts for treatment and dose and includes all the biomarkers of interest

cph(Surv(time, event) ~ treatment + dose + treatment*dose + biomarker1 + biomarker2 ...)

I'm aware that in the development of prediction models, overfitting is a major concern as too many variables can cause overfitting and prevent the model from validating. As I'm interested in effects estimation to understand the data at hand, not building a prediction model, do I need to be concerned about overfitting and whether I should limit the number of included variables? If so, how should I determine how many covariates to include?

Do I need to limit the number of variables included in my model, or can I not worry about overfitting and use my full model with all treatment effects and biomarkers, understanding that the small N may cause my estimates to not replicate in future larger studies?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

You did not state the all-important number of events, but clearly it cannot exceed 20.

There is a lower limit of sample size required to estimate even the simplest thing, such as an overall event incidence. If there were no censoring, the sample size required to estimate a simple proportion is 96. That is just to estimate the intercept in a binary logistic model. To have covariates you would clearly need more than 96 subjects. Any analysis of 20 subjects is futile.

A recent paper showing how to calculate sample size when covariates are present is here.

For information about where 96 comes from see BBR and RMS in these links.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer Dr. Harrell. I suspected with 17 events it would be impossible to find anything. Is it safe to say then that when considering the number of covariates in a model, the sample size calculations from the paper you linked should be applied regardless of whether the model is intended for effect estimation vs prediction? $\endgroup$
    – Tomas Bencomo
    Commented Jul 14, 2019 at 19:42
  • $\begingroup$ That's mainly right. I have another calculation in my RMS course notes that shows how many observations are needed in a Cox model just to estimate the underlying survival curves with no covariates. It's something like 150. So I don't think the dataset of 20 persons can be used for anything unless you had a single fully pre-specified risk factor to test for correlation with event time. $\endgroup$
    – Frank Harrell
    Commented Jul 15, 2019 at 22:30
  • 1
    $\begingroup$ I could see at the end of the recommended paper that was in development tools for estimating sample size / covariates. I am finding the same problem in a case-cohort study (Cox regression as well). I am asked to adjust for multiple covariates, that I don't think will be require (9 predictor variables with 196 controles and 55 cases). I don't know how to detect overfitting or even less the sample size needed for this number of covariates. Thank you again for you contribution Prof Harrell $\endgroup$
    – Javier Hernando
    Commented Apr 24 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.