0
$\begingroup$

I am doing research for a data science class and am looking into the benefits of storing all unique words in a document along with the entire document.

I am looking for some research that has been published that averages the number of unique words in a document against the total number of words in a document.

I want to evaluate the added storage requirements when storing a list of unique words along with the original document versus the reduced search space when searching for words in a document.

$\endgroup$
6
  • $\begingroup$ It ought to depend on the type of document and document length. Why not examine a sample of the documents you intend to process? $\endgroup$ – whuber Mar 21 '18 at 13:28
  • $\begingroup$ This was for a general case of a data storage where anybody could upload documents so samples cannot be acquired $\endgroup$ – Thor Mar 21 '18 at 16:48
  • $\begingroup$ Then you have no hope of obtaining any objective or well-supported answer: you will have to guess. $\endgroup$ – whuber Mar 21 '18 at 17:07
  • $\begingroup$ The information I was looking has been accepted as an answer $\endgroup$ – Thor Mar 21 '18 at 17:51
  • 1
    $\begingroup$ To begin with, I was hoping for a average number of "all documents". And you are absolutely right, without a sample I won't be able to get the true search space reduction but this is the closes I can get. I am going to make some sample sizes and try to show how the search space grows slower using unique words rather than the entire document as the document size grows. $\endgroup$ – Thor Mar 21 '18 at 20:03
0
$\begingroup$

Heap's law says the number of unique words in a document obeys a power law in the length of the document. For a document of length $n$, the number of unique words is $V_R(n)=Kn^\beta$ for some $K$ and $\beta$. https://en.wikipedia.org/wiki/Heaps%27_law

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.