1
$\begingroup$

I am undertaking a pilot project using the first 62 data points I have manged to collect. Currently I have 26 cases (disease positive) to 36 controls (disease free).

I have used Firth's Logistic regression to examine the relationship between several radiographic independent variables and the disease*. I've selected Firth's in light of the small sample size.

However, now I want to test the variables of interest by adjusting them for the usual things; age, gender, kidney function etc...

How many covariates can I add to my model?

If I am understanding this paper correctly, it would seem, I can add quite a few!

But what would be 'best' practice here? Would it be better to try creating a series of 2 variable models, testing my IV of interest with one covariate at a time? Or load up the maximum ?

**I am only looking to identify significant relationships between the disease and radiographic IVs. I'm less interested about getting the most precise effect estimates, its more about which IVs are significant and their direction.

Edit: just thought I would add some output. Confidence intervals are very large!


                  coef    std err     [0.025    0.975]      p-value
---------      ---------  ---------  ---------  --------  -----------
X-ray3          1.62881    0.450813   0.839258  2.63676   0.000003
Intercept      -0.197515   0.303736  -0.801548  0.392597  0.4985

Log-Likelihood: -29.5737

conf-interval:               Odds ratio
X-ray3    [2.31 - 13.96]     [5.255]
Intercept [0.44 - 1.481]




              coef       std err     [0.025      0.975]      p-value
---------  ----------  ---------  ----------  ---------  -----------
X-ray3      1.65909    0.459325    0.859825   2.70772    0.000002
Weight     -0.0529905  0.0585941  -0.177383   0.0537212  0.334079
Age        -0.0289818  0.0271007  -0.0833444  0.0217849  0.258607
Sex         -0.302079   0.65731    -1.60463    0.950265   0.531984
Intercept   3.12534    2.44046    -1.36478    8.22448    0.170023

Log-Likelihood: -21.5074

conf-interval:               Odds ratio
 X-ray3    [2.36 - 14.9]     [5.255]
 Weight    [0.83 - 1.05]     [0.948]
 Age       [0.92 - 1.02]     [0.971]
 Sex       [0.20 - 2.58]     [0.739]
 Intercept [0.25 - 3,731.17]

```
Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Your situation is discussed in "Too many covariates and too few cases? – a comparative study," by Chen, Nian, Zhu, Talbot, Griffin and Harrell, Statistics in Medicine 35: 4546-4558 (2016).

Firth regression has some disadvantages here, as it penalizes all the predictors, including your predictor of primary interest (X-ray3 here), and it also penalizes the intercept. Chen et al. suggest ridge regression penalization on all of the covariates except the predictor of primary interest. Standard procedure with ridge regression leaves the intercept unpenalized.

The usual rule of thumb for logistic regression to avoid overfitting is one unpenalized predictor per 10-20 cases in the minority class. So your current data (26 cases) should allow fitting your X-ray3 predictor unpenalized along with a (heavily) penalized set of clinical covariates. As you accumulate more cases, you can penalize the clinical covariates less.

An alternative would be to use data reduction (without looking at the outcomes) to compress your set of clinical covariates into a small number of unpenalized predictors. Harrell's course notes illustrate that approach in Section 4.7.

Omitting variables related to outcome (as needed for your proposed set of 2-variable models) is not a good idea in logistic regression, as there is an inherent omitted-variable bias even if omitted predictors are uncorrelated with those in the model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Wow, @Edm your answer is so helpful I nearly fainted! And thank you so much for including a paper and textbook. $\endgroup$
    – Maks Hall
    Commented Nov 24, 2022 at 16:28
0
$\begingroup$

It depends on what you mean by "How many covariates you can add to your model?". Firth logistic regression will run and produce some results with a very large number of covariates. That doesn't mean that the results will make a lot of sense or that the point estimates would be particularly close to what you'd get with much larger sample sizes (of course the confidence intervals you'd get would already suggest that).

The paper you cite does not say a lot of covariates would work when you only have 26 cases, the paper says that to some extent more than 2 covariates might well be okay when you use Firth (but that total sample size also matter, which is again very small for you) and that the topic is poorly understood (i.e. we can't totally be sure).

That's all before we consider that you seem to be doing model building, which adds even further instability to your results / makes any model "testing"/"validation" on the same data very questionable (in any case, that's not what pilot projects are usually intended for). The best practice here might be to collect a lot more data to assess the model (or to even build a more stable one) with a well-planned and pre-specified validation strategy.

I cannot think of a situation where trying one covariate at a time has much of a justification/logic behind it, or would be expected to do something useful.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you @Björn. The plan is for 120 cases - unfortunately it's a rare disease and data-collection is slow, but we will get there. I was also hoping to submit (in December) a very cautiously worded abstract to showcase our current study. I suppose my rationale for a 2 IV model is to show that my X-ray feature remains important when paired with e.g weight. But keeping the model complexity low - so as to keep the model ever so slightly less shaky (than it already is!). $\endgroup$
    – Maks Hall
    Commented Nov 24, 2022 at 13:40
  • $\begingroup$ The objective of the pilot is really to showcase that something might be present and useful in the X-rays, and worth following up on. Are there any guides on how to execute a pilot of this nature (I can't seem to find much that fits)? Also - might bootstrapping be a useful gauge of model stability? $\endgroup$
    – Maks Hall
    Commented Nov 24, 2022 at 14:09
  • $\begingroup$ 2 IV model keeps complexity low, but does not tell you much about what happens if you adjusted for all "known" important covariates. Yes, bootstrapping illustrates how stable (or not) the particular selected model is, but you'd want to bootstrap, do the whole model building process & then look at how stable what comes out at the end is. There's a reason why there's not much guidance on how to do this in pilot studies: they are typically for checking the data can be collected in the planned way, trying out procedures etc., not for already answering the question of the study it is a pilot for. $\endgroup$
    – Björn
    Commented Nov 24, 2022 at 17:18
  • $\begingroup$ Thank you @Björn , this is really great advice. I suppose people getting hung up on pilot conclusions is one reason some studies don't get any further? $\endgroup$
    – Maks Hall
    Commented Nov 24, 2022 at 17:33
  • $\begingroup$ It is important to note that for many cases the bare minimum sample size needed to estimate binary logistic model parameters when there are zero predictors (i.e., the easiest case where one need only estimate the intercept) is $n=96$ as detailed in hbiostat.org/rmsc/multivar.html. And that only achieves a mediocre marge of error in estimating the event probability (which in this case is crude prevalence of the outcome). $\endgroup$
    – Frank Harrell
    Commented Jul 16, 2023 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.