I have training data set with around 1500 positive set samples and 4500 negative set samples. All the features are numeric( floating or integer type values) and the data is specific to bio-informatics domain. I have tried using SVM and random forest, both give me an accuracy of around 88% on test set of size 700 and containing only positive samples. I have also tried LDA and adaboost in R, but they give worse results. I have tried feature selection which I observed improves the prediction accuracy by another 1.5%. Algorithm used for feature selection was random forest. I have two questions:

  1. I need to improve my classification accuracy on the given test set.Please guide me how I can achieve the same.
  2. Also, if there has to some step wise strategy that goes into selecting a classification algorithm. Thanks
  • $\begingroup$ Classification accuracy of a positive only test set is a meaningless statistic. For example, a classifier that always returns 'positive' will perform at 100% accuracy. $\endgroup$ – Trisoloriansunscreen May 5 '14 at 15:30
  • $\begingroup$ got your point @Tal :) $\endgroup$ – priyanka May 5 '14 at 15:43

One thing is that your training data ought to be similar to your test data. You say your training data is 25% positives and your test data is 100% positive. That's not good. In fact, having test data that is 100% positive seems wrong to me.

Training data and test data should be randomly split from your full data set.

Once you've fixed that, you may have better accuracy.

However, it may also be that there is no better model. 88% accuracy is pretty good in many situations; why are you sure that you can do better?

  • $\begingroup$ Thanks for the answer. Currently,I am just exploring means of improving my answer. Additionally, the real data is big data and in real data set only 1% of values are positive, but when i test it on sample set(whose true identity is unknown) its around 6%. which is no good in case of big data. $\endgroup$ – priyanka May 5 '14 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.