1
$\begingroup$

The caret package offers the ability to make statistical statements about the performance of different models used for classification. According to the description, it uses asymptotic simultaneous confidence intervals for many-to-one comparisons of proportions. I have read the papers by Horthon et al (2005) and by Eugster et al. 2008 but yet still I cannot summarize what the results mean nor how it was arrived at. I am trying to interpret this graph (taken from the caret page. enter image description here

Can anyone offer a brief description on an interpretation of the graph, especially the x-axis scale (-0.4 to 0.6).

Any comments will be welcomed. Kurt

$\endgroup$

1 Answer 1

2
$\begingroup$

Code is always helpful but I think you are looking at the difference in the area under the ROC curve (in the left panel) between two models. Since the models used the same resamples, you have paired observations:

 resample   model1_ROC   model2_ROC  difference
        1         0.95         0.90        0.05
        2         0.70         0.75       -0.05
        3         0.80         0.81       -0.01 
        :          :            :           :

The plots are showing the distributions of the paired differences (I'm assuming you used bwplot.diff.resamples). Inference is done of the pairwise differences between models but the references have other ways of looking at them (with mixed models I think).

Max

$\endgroup$
2
  • $\begingroup$ Thanks Max for the response and the great package.Your answer has started to make things a bit clearer for me. So I would assume that the x-axis is the difference between the 2 models? $\endgroup$ May 13, 2014 at 21:58
  • $\begingroup$ Yes, you are correct; it is the difference in metrics between models. $\endgroup$
    – topepo
    May 14, 2014 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.