Given probability x, how likely is it that probability y is due to chance? I wrote a Python script searching text for words and separating unique and non unique words.  Running the pages I have through my program, I find 1041 unique words out of 3742 total, so roughly 27% of all the words are unique.  Yet when I look at the first word of each page, 33 of the 44 words starting each page are unique, for a 75% ratio.  
How do I measure how likely this 75% result is due to chance, given the 'normal' 27% unique word probability? I have some experience in R if that helps at all in explaining things to me. 
 A: I think that the term 'by chance' is not clearly defined as long as you do not have a specific hypothesis you want to test. 
You could regard the full text as your population. The complete 'census' of all words resulted in the 'true' parameter $\theta=.27$, say. 
Now you describe that you took a 'sample' of words, whose characteristic is page position (first word on each page) and you want to test the hypothesis, whether page position affects the probability of a word being unique. 
Hence you want to test: $$H_0:\theta=.27$$ which is equivalent to asking whether the sample of words comes from the population of all words (your full text) or forms an own (sub-) population. 
If we regard the 44 pages (words) as independent draws from a Bernoulli distribution, the number of positive outcomes $X$ is Binomial. Now we need 
$$P(X \ge 33|H_0) \approx 4.68*10^{-11}$$ 
As you can verify using R pbinom(32,44,.27,lower.tail=FALSE). This probability is very small, so you can say with very low probability of error that observing 33 unique of 44 words was not caused by chance, because if the null hypothesis was true the pobability of this event happening by chane alone would be very small. Hence, $\theta$ of the sub-population of words at the top of all pages seems to be different from your population $\theta$ of .27. 
Put differently, position seems to have an impact on the probability of uniqueness. Only in a very small proportion of cases you would make an error when claiming this.
