Hi so I'm trying to implement binary classification of scientific publications from various journals such as bioinformatics, nature etc. The goal is to classify each publication as either a software tool or a non tool. A software tool is defined as any publication which has some open source code/implementation available anywhere online and hence has a link for that in the publication text.

My training set is around 60 tools and 170 non tools consisting of the entire text of the publications (from various journals). What is the best approach to solve this problem and obtain around ~90% accuracy? Here's what I've tried so far using scikit-learn:

  1. Used TFIDFVectorizer on entire text and then SVM with grid search to find optimal parameters. Couldn't achieve more than 75-80% accuracy.

  2. Only used sentences which contained urls (for both training and testing) and used tf-idf with LSA(truncated svd). Used SVM, SGDC classifiers but again same accuracy as above on average.

The main problem I'm facing is that after many different approaches most tools still get misclassified as non tools. Many publications themselves make use of some software tools but are still not software tools themselves. So I decided to focus on the urls rather than the entire text but am still not getting the accuracy I want.

Any suggestions/approach on the problem would be greatly appreciated.


1 Answer 1

  • Are you sure you even need machine-learning process for that kind of task? I suggest that a classic text analyzer catching all the URL containing "github" or "gitlab" could be just fine for what you want to do. Maybe setting up a list of strings that if present in the paper indicate you that it's a tool.

  • Your dataset is way too small to train a classification model. I'm even surprised your reaching 75% of accuracy. I don't know if you have the possibility to get more labelled data but 1000-2000 entries would be a good start to effectively train your model.

  • 1
    $\begingroup$ Regarding the accuracy: a trivial model that says everything is a non-tool has $170/230 \approx 74$% accuracy given the class distribution mentioned in OP. Just shows accuracy is a very bad metric that should be avoided. $\endgroup$ Aug 25, 2016 at 13:47
  • $\begingroup$ Yeah good catch, I assumed it was a 50/50 class repartition my bad. But yeah recall and precision are far more reliable in my opinion. $\endgroup$ Aug 25, 2016 at 13:52
  • $\begingroup$ Good points and thanks for your input. However, only "github" and "gitlab" etc will not work since a lot of the times the implementation is available on domain names of some university and I can't rely on it being a popular code sharing repository. $\endgroup$ Aug 25, 2016 at 17:47
  • $\begingroup$ And yes, I'm looking at the confusion matrix which is basically telling me that mostly everything is getting classified as a non tool. If I gather more data as you suggested, what would be a good machine learning model for this task? $\endgroup$ Aug 25, 2016 at 17:48
  • $\begingroup$ And also even if github etc is present in a url, it does not necessarily mean that the publication itself is about a tool. Many non tool publications are about using some software tools so they credit the tools properly and might use their github links in the text. $\endgroup$ Aug 25, 2016 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.