1
$\begingroup$

I am doing classification of "text quality" using four classes and using 30 features with 1300 samples. I am using the following classifiers:

  1. LDAC based on linear discriminant analysis from mlpy.
  2. svm with rbf and gridsearch
  3. svm with polynomial kernel degree 5 and gridsearch and
  4. random forest.

The svm and random forest classifiers are based on scikit-learn.

I am getting similar (poor) accuracy results with all the classifiers. I get approx 57.1% with LDAC, svm and random forests vary from 57.8- 58.1%. The accuracy is based on cross validation and the corresponding confusion matrices (sum of diagonal/total).

Why am I getting similar results with all the classifiers? Any suggestions about the reasons for having similar results when using linear and non-linear classfiers? Is this agreement between linear/non-linear classifiers at low-level a strong hint of flaws in some specific area? I appreciate any comments here in order to try to improve this situation!

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ It's possible that your features just aren't strongly related to your classes, at least in this sample, and that the relationship that does exist is well captured by the linear methods. $\endgroup$
    – Peter Flom
    Commented Nov 13, 2012 at 11:16
  • $\begingroup$ If you have four classes, guessing will give you 25% accuracy as a baseline. Thus, if you get up to nearly 60%, I would not call that poor. $\endgroup$
    – bayerj
    Commented Nov 13, 2012 at 19:39
  • $\begingroup$ Whether the results are useful or not depends on the distribution of your outcome. If the most common class occurs ~55-60% then your results are useless. If on the other hand the classes are uniformly distributed (25% each), then 57-58% is not bad. $\endgroup$
    – George
    Commented Nov 21, 2015 at 12:30
  • 1
    $\begingroup$ How are you defining poor? What baseline are you using? Is there some reference baseline from the literature? Are you comparing it to a simple method like k-means, etc? Also have you tried doing any data pre-processing like zero-centering or normalization? $\endgroup$
    – Indie AI
    Commented Dec 21, 2015 at 18:04

1 Answer 1

Reset to default
1
$\begingroup$

First of all you need to understand whether your features are really doing their job and help separate your classes. For this sake many instruments are available, take a look at Orange ( http://www.biolab.si/supp/bi-vizrank/gui.htmu ) just to name a few.

Typically, these instruments will hint you about most correct classifier. Without seeing your data any conclusions are very uncertain.

And yes, if you indeed have only two classes, having 60% accuracy is not so bad on its own.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.