2
$\begingroup$

I want to know how to get several performance measurements of a generated WEKA model. Note that I am predicting a two-class variable, Alive or Dead and I use the developer version 3.7.10 in my own Java project.

  • Confusion matrix

    I know this is a simple question but am I correct that the one below is a direct equivalent of a traditional confusion matrix?

    === Confusion Matrix ===
    
         a     b   <-- classified as
     13735   216 |     a = Alive
       392   657 |     b = Dead
    

    As in:

    TP | FN
    FP | TN
    
  • Sensitivity and specificity

    Given a part of the WEKA result buffer below, below contains the ROC, specificity (or Recall) and sensitivity (or Precision) measurements but depending on a given class.

    === Detailed Accuracy By Class ===
    
                     TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                     0.985    0.374    0.972      0.985    0.978      0.665    0.974     0.998     Alive
                     0.626    0.015    0.753      0.626    0.684      0.665    0.974     0.775     Dead
    Weighted Avg.    0.959    0.349    0.957      0.959    0.958      0.665    0.974     0.982     
    

    Poor Felix asked the questions "WEKA Specificity and Sensitivity for global Rule Model" in (a), (b) and (c) similar to the one I have right now but still didn't get the exact answer.

    Now, is it right to interpret that the overall ROC, specificity and sensitivity of the whole model are the ones in the Weighted Avg. row?

$\endgroup$
3
$\begingroup$

Your description of the confusion matrix is correct assuming alive people are defined as a positive outcome. Those entries are the correct order.

TP | FN
FP | TN

I do not like how Weka labels the columns. TP Rate (for example) is based on that row being the positive. So the second entry under TP Rate (0.626) is actually the TN Rate. The other columns are defined similarly. For example: The second entry under FP Rate (0.015) is actually the FN rate. The second entry under Precision (0.753) is the "Precision" of the negatives (aka negative predictive value). In other words, the column labels are only correct for the first row assuming the first row is defined to be the positive outcome.

The last row "Weigthed Avg." is not necessary for binary classification. It is weighting the results based on the sample sizes for each class. For example the last row first column, TP Rate (0.959), is actually the overall accuracy of the model. I would ignore this last row.

$\endgroup$
  • 1
    $\begingroup$ (1) I agree, the way the labels are placed confuses me. (2) Now I understand the parts pf TP Rate and others, thank you for that. From your answer, will it be safer for me to just derive precision (= TN/(TP+FP)) and recall (= TP/(TP+FN)) from the generated confusion matrix? (3) Thank you for the TP Rate - Overall accuracy correlation. I am correct! $\endgroup$ – Saggy Manatee And Swan Folk Jan 31 '14 at 18:57
  • 2
    $\begingroup$ Honestly, I would just do it by hand using the equations you provided just to double check (or at least "eyeball" the table values to make sure they represent what you think they do). The Weka output is somewhat obscure and gives me a headache every time I look at it. $\endgroup$ – Underminer Jan 31 '14 at 19:03
  • 1
    $\begingroup$ I found it: the first row of values has the overall precision and recall. $\endgroup$ – Saggy Manatee And Swan Folk Jan 31 '14 at 19:11
  • 1
    $\begingroup$ Correct. The first row has the correct column labels. The other two rows, well, you have to figure out what they actually represent on your own. $\endgroup$ – Underminer Jan 31 '14 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.