# Predict interesting articles: increase accuracy

I'm trying to write a GUI to display articles, and predict which articles I could like, based on the articles I previously liked. This post is the continuation of this one: https://stackoverflow.com/questions/26442680/predict-interesting-articles-with-scikit-learn

For now, I've written all the get-data part, i.e I can parse the RSS web page of some famous scientific journals. The GUI is almost done as well.

So, I populated a database of articles, with some infos (it's a sqlite database): title, authors, abstract, etc: https://www.dropbox.com/s/03pkkm9g4x3y1i6/fichiers?dl=0

Some articles in this database are liked, which means I think they are interesting for me. And now, based on these articles, I try to calculate a match percentage for the other articles.

To do that, I use a naive Bayes, and I give him the abstracts of the articles (it's a short text describing the article):

def initializePipeline(self):
if self.bdd is None:
self.bdd.setDatabaseName("fichiers.sqlite");
self.bdd.open()
query = QtSql.QSqlQuery("fichiers.sqlite")
query.exec_("SELECT * FROM papers")

while query.next():
record = query.record()
if type(record.value('abstract')) is str:
simple_abstract = record.value('abstract')
if type(record.value('liked')) is not int:
category = 0
else:
category = 1
self.x_train.append(simple_abstract)
self.y_train.append(category)

self.x_train = np.array(self.x_train)
self.y_train = np.array(self.y_train)

#Using a count of word, then a tf-idf transformer, and finally a Multinomial Naive Bayes
self.classifier = Pipeline([
('vectorizer', CountVectorizer(
stop_words=self.stop_words)),)),
('tfidf', TfidfTransformer()),
('clf', MultinomialNB())])
self.classifier.fit(self.x_train, self.y_train)


I give the liked articles to this classifier, for learning.

And then, for each article, I calculate a match percentage:

def calculatePercentageMatch(self, test=False):
print("Starting calculations of match percentages")
query = QtSql.QSqlQuery("fichiers.sqlite")
query.exec_("SELECT id, abstract FROM papers")
list_id = []
x_test = []

while query.next():
record = query.record()
if type(record.value('abstract')) is str:
abstract = record.value('abstract')
print(abstract)
print("\n")
list_id.append(record.value('id'))
x_test.append(abstract)

x_test = np.array(self.x_train)
list_percentages = [ round(float(100 * proba[1]), 2) for proba in self.classifier.predict_proba(x_test) ]

if test:
print(list_percentages)
else:
for id_bdd, percentage in zip(list_id, list_percentages):
request = "UPDATE papers SET percentage_match = ? WHERE id = ?"
params = (percentage, id_bdd)
query = QtSql.QSqlQuery("fichiers.sqlite")
query.prepare(request)
for value in params:
query.exec_()
print("Done calculating match percentages")


What I do is basically to calculate the probability for an article to be in one of these categories:

0: not liked
1: liked


And then I display P(liked) * 100 in the GUI. I can sort the articles with this percentage.

But, and that's where I'm going after all that, for now the percentages are ranging from 0 to 1.5 %, and the "suggestions" are not quite good. It works, but not very efficiently.

I'm looking for a way to increase the accuracy of the suggestions. Do you have any idea ?

EDIT: This question has been migrated from stackoverflow, so this is an edit to focus more on the data.

I have 1403 entries in my db. These 1403 observations are my training set, and my test set. What I want to do is to continuously analyze my test set. I "liked" 28 articles. So 28 articles are in the "1" category, and 1375 are in the "0" category.

So I train my classifier with these data: 28 articles in the 1 category, and 1375 in the 0 category. Then, with this trained classifier, I re-analyze all the 1403 articles, and I take the P(1) of each article. The goal is to spot the articles in the "0" category with a high P(1). I'll read them, and if they are interesting, I'll like them, and start over the whole process.

I'm basically trying to build a suggestion program.

EDIT 2: I edited my code. I initialize my classifier with a count of word, then a tf-idf transformer, and finally a Multinomial Naive Bayes. I also remove the stop words (it's a basic list of English stop words coming with the library I use).

• Why not use the full text instead of only the abstract? – John Zwinck Nov 9 '14 at 14:00
• Because the full text is in a pdf you have to download, and is not public. You have to pay a subscription to be able to download it, and I don't want to bypass the editor website. – Rififi Nov 9 '14 at 14:03
• Pick some articles that you do like but are not in the training set and see how your algorithm classify them. And compare with articles that you do not like. In any case, for more help here, we need more info on your dataset and less on your code. – Manoel Galdino Nov 10 '14 at 11:48
• I have to agree with @ManoelGaldino . It would be better to replace the python code blocks with description of what is being done. Or at least provide such a description side-by-side with code. Now you merely say that you are using naive Bayes but that is too broad. Also you can describe your features better - are you using counts of words as your features? If that is the case - do you remove the words like "and", "or", "the" from the abstracts? etc. – Karolis Koncevičius Nov 10 '14 at 14:20

I don't have enough space to write a comment, so I'll put an answer here, but think of it more as a guide to what kind of information would help us to help you.

Before that, however, I suggest you to take a look at this post by Peter Norvig explaining spelling correction with naive bayes.

I imagine that your model is something like:

argmax_c P(article|w1, w2, ... wn)


which is equivalent, by Bayes theorem, to:

argmax_c P(w1, w2, ... wn| article) P(article)


P(article) is your prior probability of an article being good (an article is either good or bad). and P(w1, w2, ... wn| article) is the model/likelihood, which are based on relative word frequency (actually tf-idf, I guess). Try to explain to us your model/likelihood and your prior. This will help us. And please note that, if your model predicts articles only in the range 0-1.5, maybe your prior is driving the numbers down. Last, but not least, do not use your training set to test your classifier. Use out-of-sample data.

• Ok let's go step by step. For each article, an abstract is available. The abstract is ~100 words long. I (the Pipeline, so I assume the countvectorizer) removes the stop words. So only relevant words remain. Then the TfidfTransformer performs the tf-idf transformation. At this stage, I have [(w1, frequency), ..., (wn, frequency)]. Then the Classifier, a MultinomialNB is trained: {[(w1, f1), ..., (wn, fn)] | category_article_n}. That's what I think I do. With a single command line, when the classifier is trained, I can get P(category_n) for each article. Then P*100 gives me a percentage – Rififi Nov 10 '14 at 19:31
• Actually, I think you get P(category_n| [(w1, f1), ..., (wn, fn)]) right? And it seems that you don't know what the prior P(category_n) is. Are you using fit_prior=True? Last, why are you using multinomial, if there is only two classes (liked and not-liked)? – Manoel Galdino Nov 10 '14 at 20:19
• Haha, actually I don't know plenty of things. I just followed the basic tuto (scikit-learn.org/stable/tutorial/text_analytics/…), and that's why I came here for help. I use classifier.fit(self.x_train, self.y_train) and then classifier.predict_proba(x_test). And about the NB, which one should I use ? A gaussian one ? – Rififi Nov 10 '14 at 20:46
• I just thought of a binomial NB, not multinomial. – Manoel Galdino Nov 13 '14 at 19:39

My naive method for this task would be:

Create a set S. For each journal article in your collection:

1. Tokenise author names. Add to S.
2. Tokenise title. Add to S.
3. Tokenise abstract. Add to S.
4. Porter stem everything in S.
5. Make a great big binary matrix, X, with one row for each article and a column for each feature in S.
6. Regress your binary column vector of like/dislike, Y, on X, using some general method like SVM or random forests.

Method two, cheat and use Sparrho.

• +1 for Sparrho, I didn't know it. But random forests doesn't seem to work for text classification, especially with that much fields, like abstract, authors, etc. I think trying to improve the Naive Bayes classification is a better approach, but I can be wrong. – Rififi Nov 10 '14 at 17:53

The reasons why you are not getting the desired results could be multiple.
Let us look at each of them carefully

1. Imbalance in the number of positive and negative examples. I suspect that the model is marking many words as a signal of uninteresting articles because of this. As a result, the number of words that signal interesting articles could be very less and is probably why you are seeing poor overlap percentages. As a rule of thumb, the number of positive and negative examples should be almost same. Try undersampling of negative examples and/or over-sampling of positive examples.
2. Look at the results carefully, try to score each sentence or word individually and see which sentence/word is being scored high.
3. By the end of the day, there is a limit to what a model can deliver. Remember that the model you are using is Naive Bayes, which assumes that all the words signal independently. Also, the features you are using are very basic. Once you rule out all the above points and still not seeing good results, the fault is either with the model or the features you are using.

Hope this helps
Thanks
• Hugh, this post is rather old. I kind of solved my problems a while ago. I don't use a Naive bayes anymore, now I use a SVM model. At first, the naive bayes seemed to work pretty well, but with a lot of data, the SVM model performs better. I finally managed to achieve what I wanted, and created my program. It's called ChemBrows (www.chembrows.com), and will be out pretty soon (I hope January, we are submitting it for publication). Thanks to all of you ! – Rififi Dec 26 '15 at 12:33
• Oh yeah, this post is old didn't see that :) – Vihari Piratla Dec 26 '15 at 13:00