# What is the amount of certainty in this yes/no test?

A child must learn a poem by heart. The poem has 200 lines. To test the child, the teacher asks the child to complete ten lines of the poem given the first half of the sentence. If the child gets, say, seven lines correct, the teacher assumes that the child knows 140 lines of the poem correctly.

What is the uncertainty in the teacher's guess of the child's knowledge?

In general, given $n$ things to learn, which may be either right or wrong, what uncertainty is associated with a test which tests $m$ of them?

• Is this homework? If so, it should have the homework tag. Jan 12, 2013 at 12:59
• I don't see this as learning n things and testing m, since the m lines are consecutive -- and poetry is learned consecutively. I see it as chunking the poem into n/m chunks. You pick one chunk at random. If I were the teacher, I would have more confidence if I started the child near the end of the poem, rather than the beginning. Jan 12, 2013 at 13:26
• Where does it say that the lines tested are consecutive? Jan 15, 2013 at 8:00

This question is formally similar to another one posted recently.

Let $\hat{n}$ denote the number of lines the student actually knows, and let $\theta = \hat{n}/n$ denote the true underlying fraction of the total poem that the student has memorized. Let $\hat{m}$ be the number of questions the student answers correctly when tested on $m$ lines of the poem. The challenge is to estimate $\theta$ and to provide a measure of the uncertainty in this estimate, as a function of $\hat{m}$ and $m$.

Assuming that the teacher chooses lines from the poem at random, and doesn't ask the student about the same line twice, the experimental procedure here involves random sampling without replacement. However, assuming that $m << n$, one may simplify by treating the situation as one of sampling with replacement, and still get approximately correct results. (To be more statistically careful, the teacher could allow for repeated tests of the same line.)

Assuming the teacher's selection of lines are truly random, the line-tests may be treated as a sequence of independent, identically distributed Bernoulli trials. For a given success rate $\theta$, the student's total score $\hat{m}$ has a [binomial distribution][2] with probability mass function

$f(k \mid \theta) = \Pr[\hat{m} = k \mid \theta] = {m \choose k}\, \theta^k (1-\theta)^{m-k}$

where $k \in \{0,1, \ldots, m\}$. A probability distribution over $\theta$ can be derived using Bayes' Theorem:

$f(\theta \mid k) = f(k \mid \theta)\cdot f(\theta) \, / \, \sum_j f(j \mid \theta)$

where $j = 0, 1, \ldots, n$.

Here $f(\theta)$ represents the test administrator's prior beliefs about the likely values of $\theta$, before performing your first test. Based on the set-up, it seems as if there should be no prior information used, and so one should represent beliefs with a uniform distribution: $f(\theta)=1$.

Using Bayes' Theorem together with the formula for the binomial distribution, one may compute the posterior distribution $f(\theta \mid k=\hat{m})$ of $\theta$. This will be a Beta distribution with paramaters that depend on $\hat{m}$ and $m$. One may express the uncertainty in the estimate of $\theta$ in terms of the cumulative mass function of the Beta distribution.