I couldn't find any tutorials on this on youtube or otherwise. I am validating a clinical prediction model and I have a set of predicted outcomes and the real outcome. I have built a ROC curve but I would like to build a calibration curve between the expected % and observed %. What are the steps I need to follow? Thanks in advance
In order to make calibration plot, you will have to do following steps yourself.
There is no automatic process.
1. You should have dependent/outcome variable and predictions.
2. Do visual binning of predictions. Transform --> Visual binning
3. Aggregate mean of outcome variable based on binned variable. Data --> Aggregate
4. Draw a graph using binned var on X and density on Y.
5. To Draw a line, go to Analyze --> regression --> Curve Estimation
(In step#3, I have assumed that your outcome variable is 0 or 1, and hence in this case mean will be the probability of event happening in that bin).
Full syntax and commands follow.
Creating binned variable:
DATASET ACTIVATE DataSet1.
* Visual Binning.
PRE_2. RECODE PRE_2 (MISSING=COPY) (LO THRU 0.000975873501824482=1) (LO THRU 0.00141701125412073=2) (LO THRU 0.00179052964935964=3) (LO THRU 0.00220329549806634=4) (LO THRU 0.00266698550653581=5) (LO THRU 0.00314263285616621=6) (LO THRU 0.00364267374421248=7) (LO THRU 0.00422690221=8) (LO THRU 0.00488431785475026=9) (LO THRU 0.00571507327812430=10) (LO THRU 0.00727421434905541=11) (LO THRU HI=12) (ELSE=SYSMIS) INTO Binned_pred2. VARIABLE LABELS Binned_pred2 'Predicted probability (Binned)'. FORMATS Binned_pred2 (F5.0). VALUE LABELS Binned_pred2 1 '' 2 '' 3 '' 4 '' 5 '' 6 '' 7 '' 8 '' 9 '' 10 '' 11 '' 12 ''. VARIABLE LEVEL Binned_pred2 (ORDINAL).
Calculating Density of event:
Draw a graph:
Copy these two newly created Vars to a new data set and then find duplicate and eliminate them.
Fit a line. Using curve fit. Analyze --> regression --> curve fit
* Curve Estimation.
/VARIABLES=Binned_pred2 WITH Density_outcome_SDH
/MODEL=LINEAR LOGARITHMIC INVERSE QUADRATIC CUBIC COMPOUND POWER S GROWTH EXPONENTIAL
Hope it helps.