2
$\begingroup$

When we are done in pre-processing the text (removing stop words, stemming, lower case) then creating a term document that has TF-IDF as weight.

How can I decide what correlation test to do between my terms and the target variable?? I want to know what terms are affecting my target. My target variable is continuous from 1 to 100.

For example, when I used Pearson correlation it did not making sense to me as I am not sure what is an increase or a decrease for a word used.

Edit: I did a scatter plot for the function score and TF-IDF for the term "word". Since the TF-IDF is sparse and has alot of zeros , the plot didnt seem to be linear or make sense... how can I proceed if I want to find the correlation ? enter image description here

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ (1) Why do you want to use correlation? (2) Correlation does not tell you what variables "affect" other variables. Correlation doe not imply causation. $\endgroup$ – Tim♦ Dec 17 '17 at 12:49
  • $\begingroup$ @Tim True enough, but Sara seems like statistical newbie. One step at a time. $\endgroup$ – Carl Dec 17 '17 at 13:02
  • $\begingroup$ If you want to look at the correlates with the frequency of a word-use, why not use the frequency of a word's use rather than TFIDF? $\endgroup$ – AdamO Dec 19 '17 at 22:23
1
$\begingroup$

Pearson correlation is normalized covariance. Square the Pearson correlation to get the coefficient of determination, denoted $R^2$ or $r^2$ and pronounced "R squared". $R^2$ is the proportion of the variance in a dependent variable that is predictable from the independent variable(s). In other words, $R^2$ relates an explained fraction. For example, if $R^2=0.8$, the word used would explain 80% of the variance of the thing you are correlating to.

The sign of the correlation coefficient indicates how the dependent and independent variables relate. For example, a negative correlation means that decreased word usage suggests an increase in what you are measuring.

There are likely better things to look at to do what you want to do, but this answers your question and is a good start.

Edit After seeing the plot, it may be that the correlation is not significant. It may also be that Spearman's rank correlation would not be significant. At a minimum, testing for significance of correlation should be performed.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Can you please give some insights on how can I proceed if Pearson correlation is not a good measure, because when i did the scatter plot , the variables didnt seem to be linear. Should i do some transformation to my TF-IDF weights? or change my target variable in some way? $\endgroup$ – sara Dec 19 '17 at 7:58
  • $\begingroup$ Need to know what the data is, and what you did. Otherwise, I'd just be guessing. $\endgroup$ – Carl Dec 19 '17 at 10:41
  • $\begingroup$ @sara your correlation may be linear, but from the scatterplot, it looks like one of your variables only has three answer categories. If so, you should look into Spearman correlation, but the interpretation is the same as for Pearson's. $\endgroup$ – Brett Dec 19 '17 at 19:40
  • $\begingroup$ @Brett Spearman's rank won't be ideal either. Too many tied ranks. Need to know more, to get better answers. $\endgroup$ – Carl Dec 19 '17 at 20:12
  • $\begingroup$ @Carl Actually I have a column called Tactic Description and I want to know if specific words in this column is correlated to my target variable which is a score from [1-100], For example, increase usage of word test increase the score. Thus, when i tokenized the text column and used tf-idf weight, i ended up with a matrix having alot of zeros and a few constant numbers $\endgroup$ – sara Dec 20 '17 at 5:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.