# Classification of scientific publications as software tools

Hi so I'm trying to implement binary classification of scientific publications from various journals such as bioinformatics, nature etc. The goal is to classify each publication as either a software tool or a non tool. A software tool is defined as any publication which has some open source code/implementation available anywhere online and hence has a link for that in the publication text.

My training set is around 60 tools and 170 non tools consisting of the entire text of the publications (from various journals). What is the best approach to solve this problem and obtain around ~90% accuracy? Here's what I've tried so far using scikit-learn:

1. Used TFIDFVectorizer on entire text and then SVM with grid search to find optimal parameters. Couldn't achieve more than 75-80% accuracy.

2. Only used sentences which contained urls (for both training and testing) and used tf-idf with LSA(truncated svd). Used SVM, SGDC classifiers but again same accuracy as above on average.

The main problem I'm facing is that after many different approaches most tools still get misclassified as non tools. Many publications themselves make use of some software tools but are still not software tools themselves. So I decided to focus on the urls rather than the entire text but am still not getting the accuracy I want.

Any suggestions/approach on the problem would be greatly appreciated.

• Regarding the accuracy: a trivial model that says everything is a non-tool has $170/230 \approx 74$% accuracy given the class distribution mentioned in OP. Just shows accuracy is a very bad metric that should be avoided. – Marc Claesen Aug 25 '16 at 13:47