0
$\begingroup$

I couldn't find any tutorials on this on youtube or otherwise. I am validating a clinical prediction model and I have a set of predicted outcomes and the real outcome. I have built a ROC curve but I would like to build a calibration curve between the expected % and observed %. What are the steps I need to follow? Thanks in advance

$\endgroup$
  • $\begingroup$ What does an ROC curve have to do with it? $\endgroup$ – Frank Harrell May 5 '14 at 20:13
  • $\begingroup$ In validating the model, I need the ROC curve for its discriminatory ability and the calibration curve to measure its calibration. $\endgroup$ – medictrader May 5 '14 at 21:09
  • $\begingroup$ The calibration curve is a great approach. The ROC curve is not relevant here, only the ROC area is, just because it happens to equal the concordance probability (c-index) which is a measure of pure discrimination (proportional to Wilcoxon statistic). $\endgroup$ – Frank Harrell May 5 '14 at 22:42
1
$\begingroup$

In order to make calibration plot, you will have to do following steps yourself.
There is no automatic process.
1. You should have dependent/outcome variable and predictions.
2. Do visual binning of predictions. Transform --> Visual binning
3. Aggregate mean of outcome variable based on binned variable. Data --> Aggregate
4. Draw a graph using binned var on X and density on Y.
5. To Draw a line, go to Analyze --> regression --> Curve Estimation
(In step#3, I have assumed that your outcome variable is 0 or 1, and hence in this case mean will be the probability of event happening in that bin).

Full syntax and commands follow.

Creating binned variable:
DATASET ACTIVATE DataSet1.
* Visual Binning.
PRE_2. RECODE PRE_2 (MISSING=COPY) (LO THRU 0.000975873501824482=1) (LO THRU 0.00141701125412073=2) (LO THRU 0.00179052964935964=3) (LO THRU 0.00220329549806634=4) (LO THRU 0.00266698550653581=5) (LO THRU 0.00314263285616621=6) (LO THRU 0.00364267374421248=7) (LO THRU 0.00422690221=8) (LO THRU 0.00488431785475026=9) (LO THRU 0.00571507327812430=10) (LO THRU 0.00727421434905541=11) (LO THRU HI=12) (ELSE=SYSMIS) INTO Binned_pred2. VARIABLE LABELS Binned_pred2 'Predicted probability (Binned)'. FORMATS Binned_pred2 (F5.0). VALUE LABELS Binned_pred2 1 '' 2 '' 3 '' 4 '' 5 '' 6 '' 7 '' 8 '' 9 '' 10 '' 11 '' 12 ''. VARIABLE LEVEL Binned_pred2 (ORDINAL).
EXECUTE.

Calculating Density of event:
AGGREGATE
/OUTFILE=
MODE=ADDVARIABLES
/BREAK=Binned_pred2
/outcome_SDH_mean_1=MEAN(outcome_SDH).

Draw a graph:
Copy these two newly created Vars to a new data set and then find duplicate and eliminate them.
Fit a line. Using curve fit. Analyze --> regression --> curve fit
* Curve Estimation.
TSET NEWVAR=NONE.
CURVEFIT
/VARIABLES=Binned_pred2 WITH Density_outcome_SDH
/CONSTANT
/MODEL=LINEAR LOGARITHMIC INVERSE QUADRATIC CUBIC COMPOUND POWER S GROWTH EXPONENTIAL
/PRINT ANOVA
/PLOT FIT.

Hope it helps.
Thanks!

$\endgroup$
  • $\begingroup$ please improve the formmating of the question $\endgroup$ – Antoine Sep 30 '15 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.