0
$\begingroup$

I am trying to create risk prediction model in R. I am new to logistic regression risk prediction analysis. I obtained reliability curve using Cal_curve <- calibrate(Multi_model_1, method='boot', B=1000) and the results are (n=2813 Mean absolute error=0.011 Mean squared error=0.00063 0.9 Quantile of absolute error=0.022). But the reliability curve looks poorly calibrated as in the attached image.

The model internal validation looks better.

validate(Multi_model_1, B=1000)

enter image description here

Sample (approximately 80 events and 2730 non-events for training datasets, 6 independent variables) I used bootstrapped internal validation with all the available sample. Is it right to split the data for testing set?

Is my model good to use or some other techniques can be applied to get better reliability curve for this logistic model?

Confused how to proceed further.

enter image description here

Improve this question
$\endgroup$
3
  • $\begingroup$ What is the “actual probability”? $\endgroup$
    – Dave
    Aug 11, 2020 at 1:26
  • $\begingroup$ @Dave OP is using Frank Harrell's rms package. You can likely learn more from examining the documentation than OP could tell you. No offense OP $\endgroup$
    – Demetri Pananos
    Aug 11, 2020 at 2:05
  • $\begingroup$ @Dave. These are my probabilities Age_bin=2 0.53 Age_bin=3 0.65 Age_bin=4 0.62 A 0.65 B 0.52 C 1.00 D=1 0.67 $\endgroup$
    – Vasanth
    Aug 11, 2020 at 2:19

1 Answer 1

Reset to default
1
$\begingroup$

Is my model good to use

Maybe. Calibration is a good thing to have since your risk estimates are meaningful. Discrimination is nice because you give people with the outcome a larger probability than those that do not, but the risk estimate itself seems to be quite dubious around the 0.5 mark, which is sort of a magic number in these sorts of things.

You're clearly using the rms package. Have you followed Frank's suggested approaches for modelling?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you Demetri $\endgroup$
    – Vasanth
    Aug 11, 2020 at 2:20
  • $\begingroup$ Yes I am using rms package. I have started reading Frank's book. If I use Bw =TRUE, it eliminates first (Age bins factor) and my reliability curve looks similar as before. Now i have 4 "x" variables. The thing is one of my "x" variable has high odds (1.85 = A 1.11 = B 171369.21 =C 2.05 = D ). Higher odd variable may be driving my model. $\endgroup$
    – Vasanth
    Aug 11, 2020 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.