I am testing various classification schemes on a training set with about 3000 instances and 20 attributes. The train set is distributed into 6 classes such that the chance accuracy would be about 18%. I also have and a test set of 500 instances. The data is clean and scaled and I tried these schemes with and without outliers.

Out of the 20 attributes I select the most relevant ones suggested by caret::rfe for each scheme and then caret::train the models to determine the best parameters. I extract bootstrap accuracy from the train function and compare it with accuracy on the test set. Here are the results

Scheme       | Bootstrap Accuracy | Test Set Accuracy
------------   ------------------   -----------------
SVM (C-svc)  |         30%        |         31%
C50          |         85%        |         26%
RandomForest |         14%        |         41%
MDA(earth)   |         25%        |         28%
multinom     |         18%        |         29%
glmnet       |         23%        |         26%

My question is how much improvement should one expect from such models since all these models fare much better than the chance accuracy (on train set), but still worse than flip of a coin on the test set.

Further what can I do to improve the accuracy of my models?

  • 1
    $\begingroup$ What happens if you omit variable selection? Also, I would worry that my test set is v. different from the training set. $\endgroup$
    – January
    Commented Sep 13, 2013 at 14:22
  • 1
    $\begingroup$ The accuracies go down moderately if do not omit variables. Are there any tests for checking if my test data is different from train set? $\endgroup$
    – earthlink
    Commented Sep 13, 2013 at 15:19
  • 1
    $\begingroup$ OK, one more thing to ask. What do you mean by "accuracy"? Percentage of correct answers, right? So, for example, RF perform much better on the test set than on the training set, correct? Did you look at the contingency tables -- maybe some classes get a particularly poor classification (often the case if they are similar). As for the second question, I'd try a PCA on test+train pooled together and just a simple t test. $\endgroup$
    – January
    Commented Sep 13, 2013 at 19:40
  • $\begingroup$ @January thank you. Yes, by accuracy I mean correctly predicted classes. I will look at the poorly predicted classes to check. I am not familiar with the test you suggested, can you suggest a reference where I can read more. $\endgroup$
    – earthlink
    Commented Sep 13, 2013 at 23:49
  • $\begingroup$ By 'accuracy' you mean unweighted accuracy, also called 'macroaccuracy'. @January is suggesting you show us the confusion matrix so we can see if any classes do much worse than others. That might inform your feature selection, or choice of classifiers (e.g. add some One-Against-All classifiers for some or all classes, instead of just relying on default multiclass classification, it might not be optimal.) $\endgroup$
    – smci
    Commented Sep 22, 2016 at 1:15

1 Answer 1

  • to start with, for 3k instances, 6 classes is an overkill. Are you sure you need 6 ? i have had experiences where i used 40k+ records with only 4 labels/classes and ended up with rather similar 30-40% accuracy. But in any case it also depends on how well your selected attributes unequivocally ID different records. Have you tried bagging / boosting and stacking techniques ? At times (again depends on the combination of models) the accuracy improves (i particularly find ADABoost to be quite a smart algo where it iteratively trains estimators on misclassified labels)
  • accuracy is somewhat misleading/overarching , please use f-score or precision / recall based on the problem set you are solving for. It also highlights which classes are creating problems for your accuracies.
  • also please read up on algos before you try each one on your dataset. For instance SVM's have this issue where they are great at binary classification but for multi class they use either a one vs all or one vs one while labelling. Not implying that they don't work but they have issues like selecting the right C values and the kernels (that are essential for multiclass/hi D problems).. you could consider neural nets and simple decision trees (again here ensure the max depth is limited) as well ..... hope this helps

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.