I have a very simple question. I am doing a simple frequency analysis where I am looking at two documents and comparing the frequency of the top used words in each document. I want to do a comparison of the two documents in R, but the problem is that one document has 10x more words than the other document. What would be the best way to normalize so that I have two side by side frequency bar graphs that allow me to compare the most frequent terms. I hope that makes sense. No need to provide code. I can figure out the code I just want to know how to present the bar graphs. I was thinking (frequency of word x)/(total number of words in document y).
Thanks!
$\begingroup$
$\endgroup$
1
-
$\begingroup$ practically you are thinking correct ! $\endgroup$– Ram SharmaCommented Jun 16, 2014 at 17:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
I think, as you mention, the frequency of a particular word divided by the total number of words in the document is a very reasonable way to standardize. For presentation purposes, you may want to multiply that proportion by some fixed number (like 1000) so that it's easier to process (i.e. comparing 4.6 vs. 5.8 instead of 0.00046 vs. 0.00058).
Also, I'd recommend stemming all the words in the document, if you haven't done so already (so words like "run", "running", etc. all are grouped together). R has some great functionality for this, too.
answered Jun 16, 2014 at 17:52