Predict interesting articles: increase accuracy

Question

I'm trying to write a GUI to display articles, and predict which articles I could like, based on the articles I previously liked. This post is the continuation of this one: https://stackoverflow.com/questions/26442680/predict-interesting-articles-with-scikit-learn

For now, I've written all the get-data part, i.e I can parse the RSS web page of some famous scientific journals. The GUI is almost done as well.

So, I populated a database of articles, with some infos (it's a sqlite database): title, authors, abstract, etc: https://www.dropbox.com/s/03pkkm9g4x3y1i6/fichiers?dl=0

Some articles in this database are liked, which means I think they are interesting for me. And now, based on these articles, I try to calculate a match percentage for the other articles.

To do that, I use a naive Bayes, and I give him the abstracts of the articles (it's a short text describing the article):

def initializePipeline(self):
    if self.bdd is None:
        self.bdd = QtSql.QSqlDatabase.addDatabase("QSQLITE");
        self.bdd.setDatabaseName("fichiers.sqlite");
        self.bdd.open()
    query = QtSql.QSqlQuery("fichiers.sqlite")
    query.exec_("SELECT * FROM papers")

    while query.next():
        record = query.record()
        if type(record.value('abstract')) is str:
            simple_abstract = record.value('abstract')
        if type(record.value('liked')) is not int:
            category = 0
        else:
            category = 1
        self.x_train.append(simple_abstract)
        self.y_train.append(category)

    self.x_train = np.array(self.x_train)
    self.y_train = np.array(self.y_train)

    #Using a count of word, then a tf-idf transformer, and finally a Multinomial Naive Bayes
    self.classifier = Pipeline([
        ('vectorizer', CountVectorizer(
                stop_words=self.stop_words)),)),
        ('tfidf', TfidfTransformer()),
        ('clf', MultinomialNB())])
    self.classifier.fit(self.x_train, self.y_train)

I give the liked articles to this classifier, for learning.

And then, for each article, I calculate a match percentage:

def calculatePercentageMatch(self, test=False):
    print("Starting calculations of match percentages")
    query = QtSql.QSqlQuery("fichiers.sqlite")
    query.exec_("SELECT id, abstract FROM papers")
    list_id = []
    x_test = []

    while query.next():
        record = query.record()
        if type(record.value('abstract')) is str:
            abstract = record.value('abstract')
            print(abstract)
            print("\n")
            list_id.append(record.value('id'))
            x_test.append(abstract)

    x_test = np.array(self.x_train)
    list_percentages = [ round(float(100 * proba[1]), 2) for proba in self.classifier.predict_proba(x_test) ]

    if test:
        print(list_percentages)
    else:
        for id_bdd, percentage in zip(list_id, list_percentages):
            request = "UPDATE papers SET percentage_match = ? WHERE id = ?"
            params = (percentage, id_bdd)
            query = QtSql.QSqlQuery("fichiers.sqlite")
            query.prepare(request)
            for value in params:
                query.addBindValue(value)
            query.exec_()
        print("Done calculating match percentages")

What I do is basically to calculate the probability for an article to be in one of these categories:

0: not liked
1: liked

And then I display P(liked) * 100 in the GUI. I can sort the articles with this percentage.

But, and that's where I'm going after all that, for now the percentages are ranging from 0 to 1.5 %, and the "suggestions" are not quite good. It works, but not very efficiently.

I'm looking for a way to increase the accuracy of the suggestions. Do you have any idea ?

EDIT: This question has been migrated from stackoverflow, so this is an edit to focus more on the data.

I have 1403 entries in my db. These 1403 observations are my training set, and my test set. What I want to do is to continuously analyze my test set. I "liked" 28 articles. So 28 articles are in the "1" category, and 1375 are in the "0" category.

So I train my classifier with these data: 28 articles in the 1 category, and 1375 in the 0 category. Then, with this trained classifier, I re-analyze all the 1403 articles, and I take the P(1) of each article. The goal is to spot the articles in the "0" category with a high P(1). I'll read them, and if they are interesting, I'll like them, and start over the whole process.

I'm basically trying to build a suggestion program.

EDIT 2: I edited my code. I initialize my classifier with a count of word, then a tf-idf transformer, and finally a Multinomial Naive Bayes. I also remove the stop words (it's a basic list of English stop words coming with the library I use).

Answer 1

I don't have enough space to write a comment, so I'll put an answer here, but think of it more as a guide to what kind of information would help us to help you.

Before that, however, I suggest you to take a look at this post by Peter Norvig explaining spelling correction with naive bayes.

I imagine that your model is something like:

argmax_c P(article|w1, w2, ... wn) 

which is equivalent, by Bayes theorem, to:

argmax_c P(w1, w2, ... wn| article) P(article)

P(article) is your prior probability of an article being good (an article is either good or bad). and P(w1, w2, ... wn| article) is the model/likelihood, which are based on relative word frequency (actually tf-idf, I guess). Try to explain to us your model/likelihood and your prior. This will help us. And please note that, if your model predicts articles only in the range 0-1.5, maybe your prior is driving the numbers down. Last, but not least, do not use your training set to test your classifier. Use out-of-sample data.

Answer 2

My naive method for this task would be:

Create a set S. For each journal article in your collection:

1. Tokenise author names. Add to S.
2. Tokenise title. Add to S.
3. Tokenise abstract. Add to S.
4. Porter stem everything in S.
5. Make a great big binary matrix, X, with one row for each article and a column for each feature in S.
6. Regress your binary column vector of like/dislike, Y, on X, using some general method like SVM or random forests.

Method two, cheat and use Sparrho.

Answer 3

The reasons why you are not getting the desired results could be multiple.
Let us look at each of them carefully

1. Imbalance in the number of positive and negative examples. I suspect that the model is marking many words as a signal of uninteresting articles because of this. As a result, the number of words that signal interesting articles could be very less and is probably why you are seeing poor overlap percentages. As a rule of thumb, the number of positive and negative examples should be almost same. Try undersampling of negative examples and/or over-sampling of positive examples.
2. Look at the results carefully, try to score each sentence or word individually and see which sentence/word is being scored high.
3. By the end of the day, there is a limit to what a model can deliver. Remember that the model you are using is Naive Bayes, which assumes that all the words signal independently. Also, the features you are using are very basic. Once you rule out all the above points and still not seeing good results, the fault is either with the model or the features you are using.
   
   Hope this helps
   Thanks