0
$\begingroup$

Consider a case where the number of labelled data as 0 = 1400 and labelled as 1 =100. The data labelled as 0 denote normal operating conditions and data labelled as 1 denote abnormal. 0 is no event and 1 is an event.

Assuming the following confusion matrix is obtained for the binary classification in Matlab's confusionmatrix() function using SVM learner

cmMatrix = 
                    predicted 0  predicted 1
           truth 0    1100 (TN)       300 (FN)
           truth 1    30  (TN)      70 (TP)


cmMatrix = [1100,300;30,70];
acc_0  = 100*(cmMatrix(1,1))/sum(cmMatrix(1,:));
acc_1  = 100*(cmMatrix(2,2))/sum(cmMatrix(2,:));

will give acc_0 = 78.5714 and acc_1 = 70

The confusion matrix is read as out of 1400 normal events, 1100 are correctly identified as normal and 300 are incorrectly identified as abnormal. Then, out of 100 abnormal events, 70 are correctly detected as abnormal whereas 30 are incorrectly detected as abnormal. I want to calculate the sensitivity and specificity with respect t class 1 since that is of primary interest in abnormal event detection. This is how I did

Sensitivity for class 1= TP/(TP+FN) = 70/(70+300) = 0.1892
Specificity for class 0= TN/(TN+FP) = 1100/(1100+30) = 0.9735

where TP with respect to class 1 = 70 FN with respect to class 1 = 300

which means that 18.92% the model will correctly identify abnormal events (with labels 1) and 3% of abnormal events will be incorrectly detected as normal events.

  • Sensitivity would refer to the test's ability to correctly detect abnormal events. Is this calculation correct. Did I do any mistake in the calculation?
$\endgroup$
1
$\begingroup$

You're correct. If a classifier is a "shoebox" and does nothing, then for $n=100$ objects (5 of which are 1 and 95 are 0), the accuracy is 95% since everything is classified as zero and only 5/100 are incorrectly classified. However, sens/spec will not be 95%. Have you tried logistic regression or using linear regression with $y=+1$ for class 1 objects and $y=-1$ for objects in class 0 (assign objects to class 1 if predicted $y>0$)?

For the SVM, there are several different kernels which can be used, like linear, polynomial, Gaussian radial basis (RBF). So try the Gaussian RBF kernel and see what happens, since it's best for data that are not straightforwardly linearly separable, but rather take on a alternating checkerboard distribution of class memberships.

Let $a$ be the true positives ($TP$), $b$ the false positives ($FP$), $c$ the false negatives ($FN$), and $d$ the true negative ($TN$). Under these definitions, sensitivity is the proportion of diseased subjects who have a positive test, defined as \begin{equation} sensitivity=\frac{a}{a+c}=\frac{TP}{TP+FN}. \end{equation} On the other hand, the specificity is the proportion of disease-free subjects with a negative test \begin{equation} specificity=\frac{d}{d+b}=\frac{TN}{TN+FP}. \end{equation}

enter image description here

$\endgroup$
  • $\begingroup$ I have used Gaussian RBF kernel. I have also calculated the individual class accuracy. My problem is that the idea of true positive(TP) is flipped in my case TP is class 1 which is the second row of my confusion matrix. In many examples, I have seen that the individual class accuracy = sensitivity?However, in my representation that is not happening. I was wondering if I have done any mistake due to the different representation. Therefore, I have posted this question. Can you please confirm if my calculation for sensitivity for class 1 is correct or not? appreciate your help. $\endgroup$ – Srishti M Jul 25 '18 at 19:50
  • $\begingroup$ Your equations for sens/spec are correct; however, your assignments are incorrect. The correct assignments are: 1100 (TN), 300 (FP), 30 (FN), 70 (TP). $\endgroup$ – JoleT Jul 25 '18 at 20:42
  • $\begingroup$ So the sens=70/100, and spec=1100/1400. FYI - the U.S. FDA demands both to be above 95% in order to approve a drug or clinical blood test. $\endgroup$ – JoleT Jul 25 '18 at 20:48
  • $\begingroup$ thank you for the comments and the update. I did incorrectly write twice TN. However, I have a concern. 300 should be FN and 30 should be FP since 300 is in the row of the negative class (0) and 30 is in the row of the positive class. In my representation would the FN and FP get flipped? $\endgroup$ – Srishti M Jul 25 '18 at 21:19
  • 1
    $\begingroup$ I'll add a pic for the way your table should be. $\endgroup$ – JoleT Jul 25 '18 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.