6
$\begingroup$

I want to predict a continuous variable from text features. Lets say I have some student essays and I want to predict their quality, as measured by a human grader, using text features (mostly words they use).

Linear regression is an obvious candidate, but if I have substantially more features than graded essays, this probably won't do well.

If I wanted to classify them into good/bad, I might try the Naive Bayes classifier. I don't, but maybe I can draw inspiration from that.

As I understand, Naive Bayes draws its power from assuming feature independence. Is there such a thing as Naive Multivariate Linear Regression, where you assume feature independence?

I think this is the same as using Univariate Linear Regression for each regression coefficient. I would expect that to run into problems quickly, though.

Is there something halfway between these two models? Putting a prior distribution on feature covariance that mostly expects conditional independence? Other models I should consider?

Bayesian models preferred.

Improve this question
$\endgroup$
3
  • $\begingroup$ I believe you are describing bag-of-words representation (en.wikipedia.org/wiki/Bag-of-words_model) as far as the regression-based (non-bayesian) model goes. One can add bigrams and trigrams to the bag to capture higher order interactions. $\endgroup$
    – Yevgeny
    Commented Jan 25, 2013 at 20:32
  • 3
    $\begingroup$ This is no criticism of your efforts, John, but it is sad to realize that you might be successful, because--by not accounting at all for word sequence, sentence structure, or underlying ideas--achieving good predictability based merely on words used would indicate how little the human grading really has to do with quality of writing or thought. $\endgroup$
    – whuber ♦
    Commented Mar 4, 2013 at 22:48
  • $\begingroup$ You may want to look into John Myles White's TextRegression package for R. $\endgroup$
    – Trey
    Commented Aug 2, 2013 at 14:41

3 Answers 3

Reset to default
2
$\begingroup$

A similar question has been asked on stackoverflow:

One answer here was to use k-nearest-neighbor regression to predict a continuous value from text documents, see https://stackoverflow.com/a/15089788/179014.

Improve this answer
$\endgroup$
2
$\begingroup$

I recommend Gradient Boosting with trees as described in chapter 10 "Boosting and Additive Trees" of The elements of statistical learning. These approach is suitable for bag-of-words-data, can catch interaction of word-features and can be used for both regression and classification.

Improve this answer
$\endgroup$
1
$\begingroup$

There is a type of Bayesian linear regression that handles the case of many features. It is called latent factor regression and you can find a good description in the paper Bayesian Factor Regression Models in the "Large p, Small n" Paradigm. If the number of latent factors is large, then it is equivalent to linear regression. Otherwise, it encourages the regression to follow the principal components of the features.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.