I came up with an idea to use machine learning for automatic grading of topic-specific texts.

More specifically, I will first use normal text classification techniques to sort all candidate texts into topics. Then, I want to be able to judge the quality of texts in specific topics:

For instance, news articles in different topics (Technology, Sports, International, Business, etc). Firstly, every article will be sorted into one topic. And what I want to archive then is to automatically grade this 'Technology' article based on the range of 1-10 or good/poor.

The criteria for such scoring scheme may be:

  • Elaboration (rich in details)
  • Coverage of all key words
  • Length?

(It is actually a little bit hard for me to quantify my criteria. The grading criteria here can be very similar to those used when human beings instinctively and quickly judge quality of two articles on a same story.)

However, as I reckon, text classification based on 'bag of words' may not be able to perform this task well. (Or will it?)

I believe such task is well-researched, and even it is a whole field rather than text classification. But I haven't figured out how to do this by searching around. Could you please refer me to related techniques/discussions/names or give me some hints if text classification can handle that.

Many thanks in advance.

  • 2
    $\begingroup$ I'm having a hard time with this question because it doesn't give sufficient clues what "quality" might mean. I would challenge the implicit assumption that humans tend to agree on their "instinctive, quick" judgments of quality. Therefore, whether this question has any answer at all comes back to the issue of quantifying "quality." $\endgroup$ – whuber Feb 3 '12 at 18:02
  • $\begingroup$ Maybe this should be moved to linguistics.stackexchange.com $\endgroup$ – Indolering Feb 6 '12 at 7:40
  • $\begingroup$ @Indolering You also do "computational" linguistics? Anyway, it is usually not acceptable to move questions to beta sites, especially old and answered. $\endgroup$ – user88 Feb 6 '12 at 11:28
  • $\begingroup$ @mbq uhh, no. I am a practitioner, kinda like most academics whom must program in order to do their job but have no business doing so. I just happen to know a lot of the background theoretical stuff that was going at the time :P $\endgroup$ – Indolering Feb 6 '12 at 18:56

There are two basic components, technical analysis (grammar, sentence length, etc) and machine learning/statistical analysis.

The best (IMHO) papers on this are those involving automated essay grading. It's a little dated, but this paper covers the techniques employed by all of the major vendors.

ETS (the people behind the TOEFL and the GRE) have put a LOT of work into this area, and they are surprisingly open about their research. The wall they run up against is that of length and specificity. They can only grade papers (with a high degree of accuracy) that are short and have specific prompts.

The biggest problem I see is that ALL of these grading facilities are based on machine learning. Even ETS uses their graders as backups for human graders; they pay 1 human to grade and if their grade doesn't match the ML grade, the essay is handed to another human grader. It essentially cuts their costs in half but it doesn't replace human graders.

If you really want to try analysis on ad-hoc text you will probably need to use the API of someone that has access to a lot of unstructured text, like OpenCalais or AlchemyAPI or even Zamenta for the categorizing and then roll your own quality analysis. However, that's a bit out of my pay range.

update Jonny's answer is really not getting enough credit. If you are really concerned about content quality, analyzing network connections and other measurements of human behavior are REALLY valuable. Again, I am no expert in this field!

It would help if you could tell us what our accuracy/validity requirements are.

  • $\begingroup$ I'm kinda nervous because all of this is based off of an ad-hoc literature review I did. I am an embodied cognitivist, so while I get the gist of how thought and language work, I only know a skiff of computational linguistics. I hope someone else can step in and validate this. $\endgroup$ – Indolering Feb 5 '12 at 2:44

More a comment than an answer, but I suspect that trying to capture the quality of a particular document in a single (non-adaptive) score is the canonical example of YMMV. For example, in a technical context users might consider quality to mean in-depth treatment of a specific problem, a survey of different techniques, position statements from experts, or any of various other measures of quality, each depending on how they want to use the information they get from the document. Furthermore, depending on the users' levels of technical sophistication, different users can be expected to rate the same documents differently, even if they have similar aims.

On the other hand this is exactly the sort of problem faced (and probably in the current state of the art partly solved) by recommender systems (Netflix ratings, Amazon recommendations, and so on). I'm no expert on these sorts of approaches, but no doubt others here are. For your problem I'd probably start my literature search in that domain.


Actually if you want to judge the quality of works, you may want to approach this with something like Google Pagerank. So for example if a particular paper is footnoted or referenced in many other papers then there is a greater chance that it is a well respected paper than a paper that is never footnoted.

Another factor which I believe would have a high correlation with quality would be the author(s). While not perfect, it is probably reasonable to assume that works by Nobel laureates within their area of expertise would tend to be of higher quality than the first work of someone who has never published before.

Third you could consider the publisher. Articles from peer reviewed journals should tend to be of higher quality than those from tabloids.

The problem with some of the items you mentioned are that they are either irrelevant to quality (length for example, by that measure the Lincoln's Gettysburg address would fail) or virtually impossible to score.

Suppose I wrote a paper about the solar system and gave the distance to the moon as 500,000 miles and said the sun was at half that distance with a temperature of 27 degrees kelvin. Would your program be able to verify or disprove those "facts"? It would score high on elaboration (lots of details, all completely wrong). As for keywords, we all know about keyword stuffing.

Some things you could easily score with a program however, such as reading level (fog index).

  • 1
    $\begingroup$ JonnyBoats here is right on. You can do a lot of tricks with NLP, but analysis of human behavior is the best way to programatically determine quality based on content. As the saying goes, "Every-time I fire a linguist..." $\endgroup$ – Indolering Feb 4 '12 at 4:11
  • $\begingroup$ Indolering: I like the Jelinek reference :-) $\endgroup$ – JonnyBoats Feb 4 '12 at 4:20
  • $\begingroup$ Jonny: Who would pass up a chance to kick Chomskian linguistics? :D $\endgroup$ – Indolering Feb 4 '12 at 4:23
  • $\begingroup$ Indolering: you might like norvig.com/chomsky.html $\endgroup$ – JonnyBoats Feb 4 '12 at 4:25
  • $\begingroup$ Jonny: Oh, that's precious. Chomsky revolutionized two field by kicking anti-theoretical linguists and behaviorists to the curb. Which was good, because it was almost impossible to get anywhere empirically without computers to crunch massive data sets. He (essentially) maintains that language can only be understood abstractly, without consideration of the rest of the cognitive system. However, you need some elaborate vision-spatial apparatus to understand why a piano can be a chair. He is correct that stats alone are not enough, but they are no different that his own abstract rule-sets. $\endgroup$ – Indolering Feb 5 '12 at 2:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.