# Randomized Results Sampling

I'm taking a look at randomized results and have a question about an end case. Specifically, suppose my scheme is as follows:

1) I want to ask question Q_A. The probability of answering Q_A in the affirmative is p.

2) I have innocuous question Q_I.

3) I have everyone in my sample flip a coin. If it comes up heads, they answer Q_A truthfully. If it comes up tails, they flip the coin AGAIN and if on the second flip it comes up heads, they answer "Yes" and if it comes up tails, they answer "NO".

4) I'm interested in estimating p.

To do this I though of the following:

$$P(Yes) = P(Yes| Coin 1= H) *P*(Heads) + P(Yes|Coin1 = T)*P(Tails) \Rightarrow$$ $$P(Yes) = p*(1/2) + (1/2)*(1/2) \Rightarrow$$ $$p = 2*(P(Yes)-1/4)$$

However, that seems to behave badly at certain points. For instance, suppose 9/10 of the class raised there hand. I then get p=1.3 which is impossible. I tried plotting the likelihood but it seems to go from 0 at -1/4 and maxes out at 3/4.

You were going along fine until you got to this point: \begin{align} P\left\{\textrm{Yes}\right\} &= p/2 + 1/4 \end{align} That gives you the probability of a "yes" for any particular subject. Now, suppose you have an experiment with $N$ subjects of whom $n$ answered "yes." What is the maximum likelihood estimator of $p$? Well, the likelihood and log likelihood functions are (ignoring unimportant combinatoric terms): \begin{align} L &= \left(p/2+1/4\right)^n\left(1-p/2-1/4\right)^{N-n}\\ lnL &= n \cdot \ln{\left(p/2+1/4\right)} + (N-n) \cdot \ln{\left(1-p/2-1/4\right)} \end{align} Maximize with respect to $p$: \begin{align} \frac{dlnL}{dp} = n \cdot \frac{1}{p/2+1/4} \frac{1}{2} + (N-n) \cdot \frac{1}{1-p/2-1/4}\frac{-1}{2}=0 \end{align} Solving . . . \begin{align} \frac{n}{p/2+1/4}&=\frac{N-n}{1-p/2-1/4} \end{align} After several algebra steps . . . \begin{align} \hat{p} &= 2\frac{n}{N}-\frac{1}{2} \end{align} This is pretty much what you got. Just as you say, the problem is that for high values of $n$, which (contrary to another answer) definitely can happen in small samples, this gives an estimator for which $\hat{p}>1$. For small enough values of $n$, you can get a $\hat{p}<0$.

So, what gives? What gives is that we have neglected to state the constraints on the maximization problem. The probability of a "yes" must lie between 0 and 1: that is $0 \le p \le 1$. This is a constraint on the maximization which we may NOT ignore (well, in large samples we can, but never mind that). In more typical derivations of maximum likelihood estimators, we just ignore the constraints and everything works out OK. This is fine if everything works out OK (that is if we find an interior maximum and the log likelihood is concave). If everything does not work out OK, then we have to pay attention to the constraints.

To be really careful, we could go back and write the maximization down as a constrained maximization. This would introduce two lagrange multipliers, one for the $p\ge0$ constraint and one for the $p\le1$ constraint. There is no need to do this, however. Since the log likelihood is concave, we can just impose the constraints ex post.

So, here is the maximum likelihood estimate:

$$\hat{p} = \left\{ \begin{array}{l} 0 \; \textrm{if} \quad 2\frac{n}{N}-\frac{1}{2} < 0\\ 1 \; \textrm{if} \quad 2\frac{n}{N}-\frac{1}{2} > 1\\ 2\frac{n}{N}-\frac{1}{2} \qquad \textrm{otherwise} \end{array} \right.$$

Obviously, the normal way of making confidence intervals won't work for cases where you are pinned against the costraints. The simplest thing to do is to bootstrap the confidence intervals.

Finally, you can use your data to test whether the subjects are following your instructions. If they are following your instructions, then $P\{\textrm{Yes}\} \ge 1/4$ and $P\{\textrm{Yes}\} \le 3/4$. These hypotheses can be tested with a likelihood ratio test using your data.

If you have the problem you are worrying about, you should suspect that the subjects are disobeying you on the coin-tossing business. In a large sample, if the subjects are obeying you, then you can't have the problem you are worrying about, except with very small probability.

There is a probability of 0.5 that the subject will get a tails on the first flip and therefor a probability of 0.5 that she/he will answer "yes" with a probability of 0.5. Alternatively there is a probability of 0.5 that she/he will answer truthfully (i.e., "yes" with a probability $p$.

$$P(yes|p) = P(HeadsOnFilp1) * P(HeadsOnFilp2) + (1-P(HeadsOnFilp1)) * p$$ $$P(yes|p) = 0.5 * 0.5 + 0.5 * p$$ $$P(yes|p) = 0.25 + \frac{p}{2}$$ $$p = 0.5 - 2 * P(yes|p)$$

Your initial sense was correct. As you've laid out the problem, it doesn't really require Bayes' rule nor any reference to innocuous question $Q_i$.

As for your scenario in which 9/10ths of the class answer yes, we can see that this is impossible. Since with $p=1$ only 75% of the class would answer "yes". Note that $P(yes)$ depends on $p$, not the other way around.

Yes, it is possible that if you ask 10 people, 9 will say "yes" under your scenario. Once you have calculated $P(yes|p)$ you can calculate the probability of 9 out of 10 yeses $P(x=9|N=10, p)$ from the binomial distribution. Moreover you can calculate the likelihood of a given $p$ with a given sample (say 9/10). Note that the likelihoods do not need to sum to unity.

So, to take an example, say $p = .8$. From the formula above we know that $P(yes) = .65$. Thus, we can compute the probability of any number $x$ of our 10 research subjects answering "yes" from the binomial distribution.

$$P(x=k) = \binom{10}{k} .65^k (1-.65)^{1-k}$$

or more generally

$$P(x=k|N, p) = \binom{B}{k} P(yes|p)^k (1-P(yes|p))^{1-k}$$ $$\mathcal{L}(p) = P(x=k|N, p) = \binom{B}{k} (\frac{1}{2}+\frac{p}{2})^k (\frac{1}{2}-\frac{p}{2})^{1-k}$$

For any value of $p$ you can calculate the likelihood $\mathcal{L}(p)$ for a given $x$ from that binomial formula. Numerically, it's easy to see that if 9/10 answer "yes" the maximum likelihood estimate for $p$ is one.

• I think your last equation should be: p= 2* P(yes|p)-0.5, (you're sign is flipped). However, this doesn't answer my question. Clearly, it's not impossible to get 9/10 to answer yes. Assume the class had 10 people. All 10 people can get tails the first time and then heads the next time yielding everyone to raise their hand. The probability of such an event is (0.5^20) > 0. Thus, I'm not quite in agreement with your last paragraph. – user1357015 Oct 7 '13 at 22:51
• Yes, I misunderstood what you were asking. I think this is what you were getting at. – Mimshot Oct 8 '13 at 16:15

The problem here is that you are merging different concepts.

One is the probability of something happens and the other is the output or the final event. In this context,the fact that 9/10 people raise the hand doesn't mean this is the actual probability that the people raise their hands, it is only an instant from the probability distribution; it is the output.

If you combine these possible outputs with the actual probabilities may be that the sum of the 'probabilities' are different from 1.