Interpretation of a WEKA result buffer - confusion matrix and performance

Question

I want to know how to get several performance measurements of a generated WEKA model. Note that I am predicting a two-class variable, Alive or Dead and I use the developer version 3.7.10 in my own Java project.

• Confusion matrix

  I know this is a simple question but am I correct that the one below is a direct equivalent of a traditional confusion matrix?

  === Confusion Matrix ===
  
       a     b   <-- classified as
   13735   216 |     a = Alive
     392   657 |     b = Dead
  

  As in:

  TP | FN
  FP | TN
  

• Sensitivity and specificity

  Given a part of the WEKA result buffer below, below contains the ROC, specificity (or Recall) and sensitivity (or Precision) measurements but depending on a given class.

  === Detailed Accuracy By Class ===
  
                   TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                   0.985    0.374    0.972      0.985    0.978      0.665    0.974     0.998     Alive
                   0.626    0.015    0.753      0.626    0.684      0.665    0.974     0.775     Dead
  Weighted Avg.    0.959    0.349    0.957      0.959    0.958      0.665    0.974     0.982     
  

  Poor Felix asked the questions "WEKA Specificity and Sensitivity for global Rule Model" in (a), (b) and (c) similar to the one I have right now but still didn't get the exact answer.

  Now, is it right to interpret that the overall ROC, specificity and sensitivity of the whole model are the ones in the Weighted Avg. row?

Answer 1

Your description of the confusion matrix is correct assuming alive people are defined as a positive outcome. Those entries are the correct order.

TP | FN
FP | TN

I do not like how Weka labels the columns. TP Rate (for example) is based on that row being the positive. So the second entry under TP Rate (0.626) is actually the TN Rate. The other columns are defined similarly. For example: The second entry under FP Rate (0.015) is actually the FN rate. The second entry under Precision (0.753) is the "Precision" of the negatives (aka negative predictive value). In other words, the column labels are only correct for the first row assuming the first row is defined to be the positive outcome.

The last row "Weigthed Avg." is not necessary for binary classification. It is weighting the results based on the sample sizes for each class. For example the last row first column, TP Rate (0.959), is actually the overall accuracy of the model. I would ignore this last row.