I'm trying to generate a predictive logistic regression model that in some form accounts for the uncertainty in the model's coefficients such that for the same input of predictor variables the outcome variable varies.

For example, if I have a model as follows:

Train <- createDataPartition(RAVE_filtered$remission_6mo_bool, p=0.60, list=FALSE)
training <- RAVE_filtered[Train, ]
testing <- RAVE_filtered[-Train, ]

ctrl <- trainControl(method = "repeatedcv",
                    number = 10,
                    savePredictions = TRUE)

summary(mod_fit <- train(factor(remission_6mo_bool) ~ `Randomized Treatment Group` + ANCA + AAV + pred_dose + baseline_BVAS,

As background:

remission_6mo_bool is a two-level factor (0,1) as is Randomized Treatment Group, ANCA, AAV and pred_dose is a continous variable, while baseline_BVAS is a discrete numerical score ranges from 0-63 with a very strong skew toward 0 (e.g., highly non-normal distribution)

The model outputs:

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3899   0.3283   0.5760   0.6516   1.0756  

                                        Estimate Std. Error z value Pr(>|z|)  
(Intercept)                            1.5419175  1.2751260   1.209   0.2266  
`\\`Randomized Treatment Group\\`RTX`  1.1282269  0.5285303   2.135   0.0328 *
ANCAPR3                               -1.3798677  0.9104766  -1.516   0.1296  
AAVMPA                                -0.3882255  0.9320723  -0.417   0.6770  
pred_dose                              0.0001030  0.0005594   0.184   0.8539  
baseline_BVAS                          0.0462253  0.0835449   0.553   0.5801  
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 114.19  on 112  degrees of freedom
Residual deviance: 106.16  on 107  degrees of freedom
AIC: 118.16

Number of Fisher Scoring iterations: 5

I'm looking for:

1) A statistically correct way on how to leverage the uncertainty of this model (or really any model) such that when used to predict outcomes it doesn't just do so formulaically such that for the same inputs the output is always the same.

2) I'm still quite new to R and statistics, so any short examples or pointers on how to implement (1) would be incredibly helpful.

Many thanks!

  • 1
    $\begingroup$ If the logistic model is completely pre-specified and all you're doing is estimating the regression coefficients, the simple confidence interval for a predicted probability seems to be what you need. $\endgroup$ – Frank Harrell Feb 2 at 16:24

If I understand your problem correctly, you want your model to predict a greater variability so that you can use this in simulations? If so, I was faced with a similar problem and the approach in this paper was useful to me:

Model Calibration Under Uncertainty - Matching Distribution Information by Swiler, Eldred and Adams.

Using this approach as a starting point, I was able to calibrate a model using minimization techniques such that the model predictions matched the observed percentiles and variability. In my case, this was more important than predicting an accurate mean value because the model outputs needed to be used as inputs into a larger forecasting model.

I hope that helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.