I'm working in R. I'd like to run a regression analysis for predicting price against terms in a text field.
I have a dataset of jewellery auction listings, with price paid, date, and an unstructured description of the item type:
text,date,price_usd "Ruby necklace, Spanish",1925,45000 "Diamond ring, 0.7 carat, bezier cut",1972,24000 "Diamond necklace",1980,87000 ...
I know how to run a linear regression for price against date:
data <- read.csv('jewels.csv') lm1 <- lm(data$price~data$date) summary(lm1)
Now what I'd like to do is build a similar model, using the words in the description field that are most associated with higher prices.
Intuitively I'd guess these include "diamond" and "necklace", while (say) "amethyst" and "ring" were associated with lower prices, but is there a way I can build a model to look at this?
My sense is that I need to do the following things:
- turn the text field into a bag of words (vector)
- remove stop words
- normalize each word for overall count(?)
- run some kind of regression against price.
I'd really welcome some guidance on how to approach each step.