I working with R on a classification problem. My outcome variable is binary with two levels 1 and 2. First of all I tried the logistic regression, which of all methods has the best performance, altough still poor.

I tried nnet package, random forest, the fuzzy package frbs and decision trees.

The nnet function gives me only one class - in this case 2.

I had some hope with frbs package. See my code below:

obj <- frbs.learn(train,method.type="FRBCS.CHI",control=list(num.labels=3,type.mf="GAUSSIAN"))
#test set without def 

But the predictions are wrong, the class 1 is completely missclassified

#percentage error
err = 100*sum(pred!=tdef)/ nrow(pred)
[1] 16.93038

I'm wondering what I could improve to classify the output variable. Is something wrong with my data? Are the parameters not right?

Can someone please verifiy? I'm at the end of my knowledge...

You can find the (normalized) data here: https://drive.google.com/open?id=1xrCXTLqKvGiGeo2X0Y1DvoSKvzbYFnyccLimceDIbZg

  • 1
    $\begingroup$ Your classes are inbalanced. Roughly 11.6% vs. 88.4%. That's why some algorithms predict only "2"s. I just split the data into 60% training and 40% testing dataset. A random forest model predicts almost exclusively "2"s and has an accuracy of 88.4% in the test dataset. Max Kuhn and Kjell Johnson discuss the problem of inbalanced classes in chapter 16 of their book Applied predictive modeling. $\endgroup$ – COOLSerdash Jul 8 '15 at 20:12
  • $\begingroup$ Did you use randomForest? How come that the accuracy is so high on the test set? $\endgroup$ – Charlotte Jul 9 '15 at 5:52
  • $\begingroup$ The accuracy is so high because almost all def in the testing set are "2"s. So predicting all "2"s gives a high accuracy. $\endgroup$ – COOLSerdash Jul 9 '15 at 5:55
  • $\begingroup$ Would it make sense to select randomly 1000 observations of class 2 and add the class 1 observations (around 600) so that the classes are more balanced ? $\endgroup$ – Charlotte Jul 9 '15 at 5:56
  • $\begingroup$ You can upsample your training dataset, to make the classes more balanced. In the book I've mentioned, they use SMOTE which can be used to resample the training dataset to make the classes more balanced. Check out the SMOTE function in the DMwR package. Another remedy could be to change the cutoffs for the ROC. Check out this blogpost by Max Kuhn. Another post that's interesting is here. $\endgroup$ – COOLSerdash Jul 9 '15 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.