Predicting continuous variables from text features

I want to predict a continuous variable from text features. Lets say I have some student essays and I want to predict their quality, as measured by a human grader, using text features (mostly words they use).

Linear regression is an obvious candidate, but if I have substantially more features than graded essays, this probably won't do well.

If I wanted to classify them into good/bad, I might try the Naive Bayes classifier. I don't, but maybe I can draw inspiration from that.

As I understand, Naive Bayes draws its power from assuming feature independence. Is there such a thing as Naive Multivariate Linear Regression, where you assume feature independence?

I think this is the same as using Univariate Linear Regression for each regression coefficient. I would expect that to run into problems quickly, though.

Is there something halfway between these two models? Putting a prior distribution on feature covariance that mostly expects conditional independence? Other models I should consider?

Bayesian models preferred.

• I believe you are describing bag-of-words representation (en.wikipedia.org/wiki/Bag-of-words_model) as far as the regression-based (non-bayesian) model goes. One can add bigrams and trigrams to the bag to capture higher order interactions. Jan 25, 2013 at 20:32
• This is no criticism of your efforts, John, but it is sad to realize that you might be successful, because--by not accounting at all for word sequence, sentence structure, or underlying ideas--achieving good predictability based merely on words used would indicate how little the human grading really has to do with quality of writing or thought.
– whuber
Mar 4, 2013 at 22:48
• You may want to look into John Myles White's TextRegression package for R.
– Trey
Aug 2, 2013 at 14:41