What train/test accuracy to expect from various classifiers on 6-multiclass problem? I am testing various classification schemes on a training set with about 3000 instances and 20 attributes. The train set is distributed into 6 classes such that the chance accuracy would be about 18%. I also have and a test set of 500 instances. The data is clean and scaled and I tried these schemes with and without outliers.
Out of the 20 attributes I select the most relevant ones suggested by caret::rfe for each scheme and then caret::train the models to determine the best parameters. I extract bootstrap accuracy from the train function and compare it with accuracy on the test set. Here are the results
Scheme       | Bootstrap Accuracy | Test Set Accuracy
------------   ------------------   -----------------
SVM (C-svc)  |         30%        |         31%
C50          |         85%        |         26%
RandomForest |         14%        |         41%
MDA(earth)   |         25%        |         28%
multinom     |         18%        |         29%
glmnet       |         23%        |         26%

My question is how much improvement should one expect from such models since all these models fare much better than the chance accuracy (on train set), but still worse than flip of a coin on the test set. 
Further what can I do to improve the accuracy of my models?
 A: *

*to start with, for 3k instances, 6 classes is an overkill. Are you sure you need 6 ? i have had experiences where i used 40k+ records with only 4 labels/classes and ended up with rather similar 30-40% accuracy. But in any case it also depends on how well your selected attributes unequivocally ID different records. Have you tried bagging / boosting and stacking techniques ? At times (again depends on the combination of models) the accuracy improves (i particularly find ADABoost to be quite a smart algo where it iteratively trains estimators on misclassified labels)

*accuracy is somewhat misleading/overarching , please use f-score or precision / recall based on the problem set you are solving for. It also highlights which classes are creating problems for your accuracies.

*also please read up on algos before you try each one on your dataset. For instance SVM's have this issue where they are great at binary classification but for multi class they use either a one vs all or one vs one while labelling. Not implying that they don't work but they have issues like selecting the right C values and the kernels (that are essential for multiclass/hi D problems).. you could consider neural nets and simple decision trees (again here ensure the max depth is limited) as well .....
hope this helps

