4
$\begingroup$

I have data in this format:

Words,Source,Result
pestology Gomel, cheap, 0
cocreating, cheap, 1
munitioner impersonating nonextinct, cheap, 1
Kolomna, expensive, 1
Enyo's snakemouth, expensive, 0
blueberries backare farriers, cheap, 0
markets rafales, cheap, 0
...

http://bit.ly/h2ynoG

How should I determine whether the phrases in column one can explain the Result value? What if I want to determine which individual words in the phrase are correlated with successful results?

Also, how should I determine whether the data in which Source is cheap is representative of all of the data?

Thanks.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ What assumptions can you make? E.g. can you assume that only one word in each phrase is responsible for the result, and the remaining words are just distractors? Or are there some "word weights" which get summed up to get the final result? Or something else? $\endgroup$
    – SheldonCooper
    Commented Mar 14, 2011 at 23:19
  • 1
    $\begingroup$ It is unknown if any of the words are meaningful. How would I determine if the words, either individually or as a phrase, are helpful in predicting the value of Result? $\endgroup$
    – cwarden
    Commented Mar 15, 2011 at 0:15

1 Answer 1

Reset to default
4
$\begingroup$

I would start by transforming the phrases into numbers via a document term matrix, with a 1 denoting the presence of a word and 0 being the absence of a word. Then you can perform correlation analysis.

http://en.wikipedia.org/wiki/Document-term_matrix

R Code

enter image description here

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks for the pointer. Could you provide an example? I installed the R tm package, and can create a DocumentTermMatrix from the Words column, but I don't know how to measure the correlation between the terms and the other variables from the data frame. $\endgroup$
    – cwarden
    Commented Mar 15, 2011 at 16:15
  • $\begingroup$ See some sample code above $\endgroup$
    – Ralph Winters
    Commented Mar 15, 2011 at 16:58
  • $\begingroup$ Looks like you've got me on the right track. I set the Result to 1 for all lines containing the word, pestology, and I get a non-zero correlation for this word: bit.ly/gtWn8c How do I determine whether the correlations between the words and Result are significantly different for the two values of Source? $\endgroup$
    – cwarden
    Commented Mar 15, 2011 at 19:35
  • $\begingroup$ You need to construct confidence intervals on the correlation estimates. stat.ethz.ch/R-manual/R-patched/library/stats/html/cor.html $\endgroup$
    – Ralph Winters
    Commented Mar 15, 2011 at 20:08
  • $\begingroup$ Do you mean perform separate correlation measurements for the cheap and expensive datasets, and then measure the correlation between these correlations, like this? bit.ly/e9DoEk $\endgroup$
    – cwarden
    Commented Mar 16, 2011 at 1:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.