0
$\begingroup$

I have a dataset with one binary class to be predicted, with 18 binary predictors and 17400 rows.

Here I used a stratified split, with approximately 85% (14648 rows) for training and 15% (2752) for testing.

The class label is very unbalanced (~ 91% of '1's and 9% of '0's).

I am using Weka to estimate a Random Forest to predict the variable, but the Kappa statistics gives me a negative result (-0.0526).

Should I discard this model (is it useful)? If not, what can I do about it, in order to produce a reliable model?

=== Summary ===

Correctly Classified Instances        2211               80.3416 %
Incorrectly Classified Instances       541               19.6584 %
Kappa statistic                         -0.0526
Mean absolute error                      0.2919
Root mean squared error                  0.4087
Relative absolute error                102.3327 %
Root relative squared error            126.1357 %
Total Number of Instances             2752     

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0,880    0,938    0,901      0,880    0,890      -0,053   0,448     0,896     0
                 0,062    0,120    0,051      0,062    0,056      -0,053   0,448     0,081     1
Weighted Avg.    0,803    0,861    0,822      0,803    0,812      -0,053   0,448     0,820     


Confusion Matrix 

  a    b   <-- classified as
2195  300 |    a = 0
241   16  |    b = 1
$\endgroup$
7
  • $\begingroup$ It depends on how it's estimated. Some people estimate it using the original scale from -1 to 1, in which case since your value is roughly 0 your accuracy is roughly 50 %. Some people use a normalized scale from 0 to 1, which is not possible here (unless there was some mistake). $\endgroup$ Dec 7, 2018 at 7:27
  • $\begingroup$ It is Weka's implementation, ranging from -1 to 1. What do you mean by 50% of accuracy? How is it related to the accuracy statistics above? $\endgroup$
    – Bruno
    Dec 7, 2018 at 8:54
  • $\begingroup$ You have an unbalanced dataset, hence a lot of your minority cases (b) were classified in a, achieving overall good accuracy but poor specific accuracies. The $\kappa$ coefficient is telling you that your model is no better than random predictions. $\endgroup$ Dec 7, 2018 at 9:36
  • $\begingroup$ Ok, so I understand that this model is useless, right? Any suggestion on what could be done to improve the classification process? Downsample/oversample the classes' instances? I tried with other classifiers, but still the same result, with kappa coefficient ranging from [-.01 to 0.02] $\endgroup$
    – Bruno
    Dec 7, 2018 at 11:24
  • $\begingroup$ You will have problems with pretty much every classifier with this data, you first need to deal with the imbalance problem. There are many posts on this site on the subject and how to deal with it. $\endgroup$ Dec 7, 2018 at 11:26

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.