2
$\begingroup$

I wrote a Python script searching text for words and separating unique and non unique words. Running the pages I have through my program, I find 1041 unique words out of 3742 total, so roughly 27% of all the words are unique. Yet when I look at the first word of each page, 33 of the 44 words starting each page are unique, for a 75% ratio.

How do I measure how likely this 75% result is due to chance, given the 'normal' 27% unique word probability? I have some experience in R if that helps at all in explaining things to me.

Improve this question
$\endgroup$
3
  • $\begingroup$ You could try modelling this process with a Bernoulli Distrbution en.wikipedia.org/wiki/Bernoulli_distribution having a success probability of 0.27. $\endgroup$
    – Paul
    Commented Dec 31, 2013 at 7:01
  • $\begingroup$ This Q&A is fairly similar: stats.stackexchange.com/questions/920/… $\endgroup$
    – Paul
    Commented Dec 31, 2013 at 7:07
  • 1
    $\begingroup$ You condition on events rather than probabilities. In this case the event you condition on is an observed proportion, not a probability. $\endgroup$
    – Glen_b
    Commented Dec 31, 2013 at 7:38

1 Answer 1

Reset to default
3
$\begingroup$

I think that the term 'by chance' is not clearly defined as long as you do not have a specific hypothesis you want to test.

You could regard the full text as your population. The complete 'census' of all words resulted in the 'true' parameter $\theta=.27$, say.

Now you describe that you took a 'sample' of words, whose characteristic is page position (first word on each page) and you want to test the hypothesis, whether page position affects the probability of a word being unique.

Hence you want to test: $$H_0:\theta=.27$$ which is equivalent to asking whether the sample of words comes from the population of all words (your full text) or forms an own (sub-) population.

If we regard the 44 pages (words) as independent draws from a Bernoulli distribution, the number of positive outcomes $X$ is Binomial. Now we need

$$P(X \ge 33|H_0) \approx 4.68*10^{-11}$$

As you can verify using R pbinom(32,44,.27,lower.tail=FALSE). This probability is very small, so you can say with very low probability of error that observing 33 unique of 44 words was not caused by chance, because if the null hypothesis was true the pobability of this event happening by chane alone would be very small. Hence, $\theta$ of the sub-population of words at the top of all pages seems to be different from your population $\theta$ of .27.

Put differently, position seems to have an impact on the probability of uniqueness. Only in a very small proportion of cases you would make an error when claiming this.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ awesome, thank you! I will start looking at the pbinom function right away. The word list I'm looking at is the Voynich manuscript, btw ... $\endgroup$
    – Rob Berkes
    Commented Dec 31, 2013 at 13:43
  • 1
    $\begingroup$ (+1) In rounding $1041/3742$ to $0.27$ you destroy all but the most significant digit in your probability(!), which should equal $1.116\times 10^{-10}$. It may also be worth noting that we should interpret your "If we regard" as an approximation to the true probability indicated by your model, which is of sampling without replacement. The sample consists of the 44 initial words out of the population of 3742 words. This lowers the answer to $8.41\times 10^{-11}$ as computed by phyper(32, 1041, 3742-1041, 44, lower.tail=FALSE). $\endgroup$
    – whuber ♦
    Commented Dec 31, 2013 at 15:01
  • $\begingroup$ I actually thought a bit about why this might happen and with what sorts of texts. Independent from the info you gave I thought that it might be a typesetting issue, in the sense that short words like 'a', 'the' (= not unique) etc often end up at the lower page and new pages are started with a longer word, which often are nouns etc, but more often unique. When writing the manuscript by hand Voynich might have done so willingly or intuitively. $\endgroup$
    – tomka
    Commented Dec 31, 2013 at 15:03
  • $\begingroup$ @whuber +1 Thanks, adds a lot! Though the p-value is changed, the conclusion would perhaps not change (p<.0001 in all cases). $\endgroup$
    – tomka
    Commented Dec 31, 2013 at 15:08
  • 1
    $\begingroup$ That is correct: changing a p-value that is already less than $10^{-10}$ by an order of magnitude will not change any conclusions. I was motivated to add that comment because it might help readers who are addressing the same problem but with different numbers; in some cases the distinctions I made will be important. $\endgroup$
    – whuber ♦
    Commented Dec 31, 2013 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.