# What is the percentage of unique words in a average document?

I am doing research for a data science class and am looking into the benefits of storing all unique words in a document along with the entire document.

I am looking for some research that has been published that averages the number of unique words in a document against the total number of words in a document.

I want to evaluate the added storage requirements when storing a list of unique words along with the original document versus the reduced search space when searching for words in a document.

• It ought to depend on the type of document and document length. Why not examine a sample of the documents you intend to process? – whuber Mar 21 '18 at 13:28
• This was for a general case of a data storage where anybody could upload documents so samples cannot be acquired – Thor Mar 21 '18 at 16:48
• Then you have no hope of obtaining any objective or well-supported answer: you will have to guess. – whuber Mar 21 '18 at 17:07
• The information I was looking has been accepted as an answer – Thor Mar 21 '18 at 17:51
• To begin with, I was hoping for a average number of "all documents". And you are absolutely right, without a sample I won't be able to get the true search space reduction but this is the closes I can get. I am going to make some sample sizes and try to show how the search space grows slower using unique words rather than the entire document as the document size grows. – Thor Mar 21 '18 at 20:03

## 1 Answer

Heap's law says the number of unique words in a document obeys a power law in the length of the document. For a document of length $n$, the number of unique words is $V_R(n)=Kn^\beta$ for some $K$ and $\beta$. https://en.wikipedia.org/wiki/Heaps%27_law