2
$\begingroup$

In short I need to know or calculate the sensitivity and specificity from Weka 3.8.4 output.

Update

I figured it out. I was confused by the confusion matrix labels. The problem I'm working on has a 'yes' class and a 'no' class. It took me ages to realize that the 'no' class is the positive class in the confusion matrix. Once I realized this I was eventually able to work things out.

I was also making mistakes because the Weka confusion matrix is a different orientation to the ones I was using as reference from other websites.

Original post:

Unfortunately I'm weak at math and am confused. I expect this will be evident in my question, which I'm finding hard to word. I find it difficult to understand equations which perform multiple actions in a single step, so I've had to make my equations step-by-step and would really appreciate the same in any answer.

For my data: True Positives = 228 and True Negatives = 104, total instances = 332

=== Confusion Matrix ===

   a   b   <-- classified as
 221   7 |   a = no
   7  97 |   b = yes

I am happy to report that I am not confused by the confusion matrix.

Sensitivity

I have read that sensitivity is the same as Recall and the same as TP Rate so I know I don't need to calculate sensitivity explicitly as Recall and TP Rate appear in the Weka output.

However I am confused by Weka's three lines of output in the Detailed Accuracy by Class (below). I can sort of see why there are three sensitivities: sensitivity to yes, sensitivity to no, and a weighted average which I don't yet understand the meaning of. I have worked out for myself that it's calculated as $$ \begin{eqnarray*} Weighted Avg Sensitvity &=&\frac{TP + TN}{TotalInstances}\\ &=&\frac{221 + 97}{332}\\ &=&\frac{318}{332}\\ &=&0.958 \end{eqnarray*} $$

However I'm confused as to what it actually means. Is the Weighted Avg TP Rate the sensitivity or is TP Rate for *yes* the sensitivity? My understanding of sensitivity is that it is all the instances that really are yes so I expect the sensitivity to be TP Rate for *yes*.

Specificity

I have done some background reading (see below) and I have the impression, possibly incorrectly, that there can be one only specificity in total. I understand specificity to mean correct identification of no. Therefore I cannot see why there would be three lots of specificity: one for no, one for yes, and a weighted average. As with sensitivity I currently have no idea what the meaning of a weighted average specificity would be. And trying to think what specificity for yes means is making my head spin.

I understand specificity to be calculated as follows: $$ \frac{TrueNeg}{TrueNeg + FalsePos} = \frac{228}{228 + 7} = \frac{228}{235} = 0.970 $$

What I really don't understand is, whether there should also be a specificity for the other two lines of output in Detailed Accuracy By Class and, if so, what those specificity figures mean.

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0.969    0.067    0.969      0.969    0.969      0.902    0.989     0.995     no
                 0.933    0.031    0.933      0.933    0.933      0.902    0.989     0.978     yes
Weighted Avg.    0.958    0.056    0.958      0.958    0.958      0.902    0.989     0.990     

=== Confusion Matrix ===

   a   b   <-- classified as
 221   7 |   a = no
   7  97 |   b = yes

I have read these three excellent explanations:

  1. Wikipedia

  2. Classeval - this guide is brilliant

  3. Lalkhen and McCluskey

but I'm still unsure whether I need to calculate specificity three times. If so, what would those equations be?

$\endgroup$

1 Answer 1

0
$\begingroup$

The problem has two classes: one called yes the other called no. However Weka does not apply semantics to the labels, they may as well be tigers and lions.

Weka labels the majority class (the bigger class) as the positive class and the minority (smaller) class as the negative class.

In this situation, the 'no' class was the larger hence it is the positive class

Once you understand that, then you can label the confusion matrix cells correctly:

TP | FN
-------
FP | TN

Weka's confusion matrix is a different orientation to the one currently on Wikipedia

Now you can do the calculations correctly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.