3
$\begingroup$

I'm currently using word_cor function (qdap package). I observed that the function is not robust as it implements Pearson, Spearman and Kendall measures only: non-occurrence of both words (in the following cross tabulation) also results in higher correlation, which should not be the case.

Cross tabulation: Cross tabulation

Code:

apply_as_df(v, word_cor, word = "hi", r=0.3, method="spearman")

Description of robustness: Correlation should not be affected by the value d - if both words don't occur in majority of documents, it doesn't mean they are highly correlated.

Is there a robust implementation of correlation/similarity (like Jaccard or cosine) for text mining in R?

$\endgroup$
  • $\begingroup$ As a hackfix, you could trim your text matrix to omit the undesired (rare) words? $\endgroup$ – shf8888 Jun 5 '15 at 14:38
  • $\begingroup$ @shf8888 : I'm inexperienced. Can you suggest edits? $\endgroup$ – Naveen Mathew Jun 6 '15 at 11:01
  • $\begingroup$ welcome. What's your goal with this robust implementation? Happy to suggest something, just want to make sure I understand your objective. It seems to me (although I could be wrong) that correlation by necessity will involve looking for patterns that involve both presence AND absence of words. However, you can control the words that are involved in calculating this correlation. For example, you could make a subset of the TF matrix that only involves the words that appear in more than X% of the documents, then calculate correlations based on this matrix. $\endgroup$ – shf8888 Jun 6 '15 at 23:33
  • $\begingroup$ @shf8888: I have implemented Kulczynski similarity measure. Let me explain with an example Doc 1: "I had a dream" Doc 2: "I had a dog" Doc 3: "I'm not a bad person" ... Doc 1000: "I'm not a bad person" Correlation between "had" and "dream": Confusion matrix: Had No Had Dream 1 0 No Dream 1 998 Correlation ~ (1+998)/1000 = 0.999 "had" occurred 2 times; it occurred with "dream" only once. Also, "dream" does not occur separately. Hence their correlation should be of the order of ~(1/2 + 1/1)/2 = 0.75. The difference is not much (0.999 vs 0.75), but it matters. $\endgroup$ – Naveen Mathew Jun 8 '15 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.