How to find words/phrases causing document classification

Question

Suppose I have a bunch of text documents. Let's say articles read by both John and Joann.

I want to kind of classify a new text document into either "preferred by John" or "preferred by Joann". Basically highlighting a bias towards one of them.

I thought of doing typical TF-IDF feature vector from the docs and running a binary classification algorithm to classify a document as either of the two classes.

However if I want to highlight those certain phrases/words in the documents which make the document have a "preference/bias" towards John or Joann, how do you think on can approach this problem.

My initial idea is to build a parametric model (like linear models etc) which can give me a weight vector corresponding to each feature in TF-IDF vector.

Then looking at the weights of these features one can say which weights cause the dependent variable (this would be John or Joann) to be either one of them. It can be a logistic classifier and then one can look at odds ratio to get the phrases/words which the algorithm thinks is causing the bias in documents.

My concern is 3-4 words might not tell if a document is biased towards John or Joann and that causes John to prefer one such article and Joann to prefer other set of articles. Also, what makes someone prefer an article might not solely be dependent upon the words/phrases but also maybe how the story or article has been written, the domain of the article etc.

Do you think this word feature weights is a good rational idea to get the bias preference sense?

Answer 1

Whether looking at 3-gram/4-gram is enough to tell if a document is biased towards John or Joann can be quantified by looking at the classification performance. Such n-grams may not capture very well the quality of writing (though they could still produce a decent approximation {1}) but it should capture the domain well.

FYI:

• Different ways to produce a confidence interval for odds ratio from logistic regression
• one can also extract the most predictive n-grams in convolutional neural networks e.g. see https://arxiv.org/abs/1703.08705 Tables 4 and 5.

---

References:

• {1} Franck Dernoncourt. "Of the use of natural dialogue to hide MCQs in serious games." RECITAL. 2012. http://francky.me/doc/TALN-JEP-RECITAL-2012-article-reviewed_en.pdf :

Early research on the automatic evaluation systems appeared fifty years ago. One of the notable projects was the Project Essay Grade, led by Ellis Batten Page at Duke University (Page, 1968). His work was based on the use of stylistic features of the response of the learner, such as word size and number of prepositions, to predict the human note of correction.