Is this method of arriving at a statistic valid?

Question

The following is an excerpt from an article about gender bias :

Overall, 34 per cent of respondents readily admitted that their family preferred having more boys. Given the sensitive nature of this question, this number is a lower bound since some people will undoubtedly give socially desirable responses.

In order to estimate the true levels of son preference, we ran a “list experiment” which protects the anonymity of the survey responses. The respondents were randomly divided into two groups, given a list of statements, and then asked the number of statements with which they agreed. Only half of respondents were given a statement about son preference, and therefore, we were able to estimate the share of respondents who agreed to the sensitive statement by comparing the means of the two respondent groups. Using this technique, we found that once granted the cloak of anonymity, 48 per cent of respondents have a preference for sons in their family.

Is this technique valid? If so, how exactly does it work? This the way I understand it (possibly wrong):

Since the respondents were randomly divided into 2 groups, it can be expected that the distribution of answers from both the groups to be identical if both the groups are given exactly the same survey. Now, say one group is given a list of 10 statements and the other is given a list of 11 statements, the extra statements being the one about gender preference. There will be a shift in the distribution and the difference gives us the percentage of people with a preference. Is my interpretation correct? If so, how is the exact percentage arrived at?

Answer 1

It sounds valid to me although you have to make some assumptions. Suppose there are $N$ respondants, and number them $i=1,2,\ldots,N$. Suppose there are ten fake questions and one real question (let's suppose the real question gets put last, so it is number eleven). Number questions $k=1,2,\ldots,K,K+1$. So, in this case $K=10$ and the real question is $k=11$. Denote the (real, truthful) answer of person $i$ to question $k$ as: \begin{align} Y_i^k &= \left\{ \begin{array}{l} 1 \; \text{agree}\\ 0 \; \text{disagree} \end{array} \right. \end{align}

Let's call the population mean of each $Y^k$, $\mu^k$. We are interested in the population mean of $Y^{K+1}$. The mean of a dummy variable is the same as the probability that it is one which is the same, here, as the proportion of people who want a boy.

Now, assume that as long as there are at least ten questions, the respondent tells the truth. Person $i$ randomized into ten questions will report $S_i = \sum_{k=1}^K Y^i_k$. The person randomized into eleven questions will report $S_i^+ = Y^{K+1}_i + \sum_{k=1}^K Y^i_k$. Let $C$ be the set of people randomized into the ten question treatment and $\#(C)$ be the number of people in set $C$. Similarly for $T$ and the eleven treatment.

If the randomization is good and under the assumption of truthful reporting, we get: \begin{align} E\left(\frac{1}{\#(C)} \sum_{i \in C}S_i \right) &= E(S) = \sum_{k=1}^K \mu^k\\ E\left(\frac{1}{\#(T)} \sum_{i \in T}S^+_i \right) &= E(S^+)= \mu^{K+1}+\sum_{k=1}^K \mu^k\\ \end{align} The first equality in each line is true by the usual arguments in RCTs. I.e. they get the expectations correct for the underlying latent variables in the treatment and control arms. And that is pretty much it, the thing we are interested in, $\mu^{K+1}$ is unbiasedly estimated by the difference in those two: \begin{align} \mu^{K+1} &= E\left( \frac{1}{\#(T)} \sum_{i \in A}S^+_i -\frac{1}{\#(C)} \sum_{i \in A}S_i\right) \end{align} There is a lot going on in the truthful reporting assumption here, though. There is the obvious "maybe ten is not enough to suppress lying" thing. But there are also the usual survey issues. Maybe the first ten questions really annoyed respondents and they were feeling disagreeable by question 11. Maybe some respondents have trouble counting, remembering where they are in the count, and thinking about answering questions (this isn't trivial: 50% of people have below average IQ). Maybe this gets worse with one extra question. If the counting mistakes are biased in one direction (and why wouldn't they be?), you can get problems. One could go on in this vein for a while.