I have a dataset with one binary class to be predicted, with 18 binary predictors and 17400 rows.
Here I used a stratified split, with approximately 85% (14648 rows) for training and 15% (2752) for testing.
The class label is very unbalanced (~ 91% of '1's and 9% of '0's).
I am using Weka to estimate a Random Forest to predict the variable, but the Kappa statistics gives me a negative result (-0.0526).
Should I discard this model (is it useful)? If not, what can I do about it, in order to produce a reliable model?
=== Summary ===
Correctly Classified Instances 2211 80.3416 %
Incorrectly Classified Instances 541 19.6584 %
Kappa statistic -0.0526
Mean absolute error 0.2919
Root mean squared error 0.4087
Relative absolute error 102.3327 %
Root relative squared error 126.1357 %
Total Number of Instances 2752
=== Detailed Accuracy By Class ===
TP Rate FP Rate Precision Recall F-Measure MCC ROC Area PRC Area Class
0,880 0,938 0,901 0,880 0,890 -0,053 0,448 0,896 0
0,062 0,120 0,051 0,062 0,056 -0,053 0,448 0,081 1
Weighted Avg. 0,803 0,861 0,822 0,803 0,812 -0,053 0,448 0,820
Confusion Matrix
a b <-- classified as
2195 300 | a = 0
241 16 | b = 1