1
$\begingroup$

I've been training SVMs over some particular data for some time. I was quite happy with the Kappa and Accuracy measures caret gives, but adding some other metrics was not a bad idea at all. The thing is whenever I add classProbs=T to the trainControl function the Cohen's Kappa is diminished in the models generated during the grid search.

I'm quite intrigued by this behavior, but I can't create a reproducible code!

With my data, the drop in the Kappa was from around 0.70 to ~0.10. When trying the same with iris I get something like:

> set.seed(101);TRAIN1<-train(Species~., data = iris, method = "svmLinear", 
+ trControl = trainControl(method = "boot", number = 10))
> set.seed(101);TRAIN2<-train(Species~., data = iris, method = "svmLinear", 
+ trControl = trainControl(method = "boot", number = 10, classProbs=T))
> set.seed(101);TRAIN3<-train(Species~., data = iris, method = "svmLinear", 
+ trControl = trainControl(method = "boot", number = 10, classProbs=TRUE))
> 
> TRAIN1$resample
        Accuracy     Kappa   Resample
    1  1.0000000 1.0000000 Resample01
    2  0.9433962 0.9148822 Resample02
    3  0.9803922 0.9705373 Resample03
    4  0.9824561 0.9731132 Resample04
    5  0.9821429 0.9727361 Resample05
    6  1.0000000 1.0000000 Resample06
    7  0.9649123 0.9470752 Resample07
    8  0.9473684 0.9211618 Resample08
    9  0.9661017 0.9489619 Resample09
    10 0.9491525 0.9233766 Resample10
    > TRAIN2$resample
    Accuracy     Kappa   Resample
1  1.0000000 1.0000000 Resample01
2  0.9433962 0.9148822 Resample02
3  0.9803922 0.9705373 Resample03
4  0.9824561 0.9731132 Resample04
5  0.9821429 0.9727361 Resample05
6  0.9807692 0.9707042 Resample06
7  0.9473684 0.9205021 Resample07
8  0.9649123 0.9473684 Resample08
9  0.9661017 0.9489619 Resample09
10 0.9661017 0.9489619 Resample10
> TRAIN3$resample
    Accuracy     Kappa   Resample
1  1.0000000 1.0000000 Resample01
2  0.9433962 0.9148822 Resample02
3  0.9803922 0.9705373 Resample03
4  0.9824561 0.9731132 Resample04
5  0.9821429 0.9727361 Resample05
6  0.9807692 0.9707042 Resample06
7  0.9473684 0.9205021 Resample07
8  0.9649123 0.9473684 Resample08
9  0.9661017 0.9489619 Resample09
10 0.9491525 0.9233766 Resample10
> 

As you can see, in the 6th and 10th resamples the results differ, but not as drastically as in my own data.

Is there any reason for that?

$\endgroup$
1
$\begingroup$

Over there at stackoverflow I've found this question (asked one day after I asked mine here, since it makes more sense to ask about statistic computation to me on crossvalidated, but whatever).

There, this other question and its answer points out there are different methods in kernlab to compute the decision when class probabilites are included. I'll requote it:

The kernlab help pages (?predict.ksvm) link to paper Probability estimates for Multi-class Classification by Pairwise Coupling by T.F. Wu, C.J. Lin, and R.C. Weng.

In section 7.3 it is said that the decisions and probabilities can differ:

...We explain why the results by probability-based and decision-value-based methods can be so distinct. For some problems, the parameters selected by δDV are quite different from those by the other five rules. In waveform, at some parameters all probability-based methods gives much higher cross validation accuracy than δDV . We observe, for example, the decision values of validation sets are in [0.73, 0.97] and [0.93, 1.02] for data in two classes; hence, all data in the validation sets are classified as in one class and the error is high. On the contrary, the probability-based methods fit the decision values by a sigmoid function, which can better separate the two classes by cutting at a decision value around 0.95. This observation shed some light on the difference between probability-based and decision-value based methods...


EDIT: Max Kuhn himself addressed this issue here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.