How to model errors around the estimation of proportions - with measurement error?

Question

I have a situation I'm trying to model. I would appreciate any ideas on how to model this, or if there are known names for such a situation.

Background:

Let's assume we have a large number of movies (M). For each movie, I'd like to know the proportion of people in the population who enjoy watching these movies. So for movie $m_1$ we'd say that $p_1$ proportion of the population would say "yes" to "did you enjoy watching this movie?" question. And the same for movie $m_j$, we'd have proportion $p_j$ (up to movie $m_M$).

We sample $n$ people, and ask each of them to say if they enjoyed watching movies $m_1, m_2, ..., m_M$ of the movies. We can now easily build estimations for $p_1, ..., p_M$ using standard point estimates, and build confidence intervals for these estimations using the standard methods (ref).

But there is a problem.

Problem: measurement error

Some of the people in the sample do not bother to answer truthfully. They instead just answer yes/no to the question regardless of their true preference. Luckily, for some sample of the M movies, we know the true proportion of people who like the movies. So let's assume that M is very large, but that for the first 100 movies (of some indexing) we know the real proportion. So we know the real values of $p_1, p_2, ..., p_{100}$, and we have their estimations $\hat p_1 , \hat p_2, ..., \hat p_{100}$. While we still want to know the confidence intervals that takes this measurement error into account for $p_{101} , p_{102}, ..., p_M$, using our estimators $\hat p_{101} , \hat p_{102}, ..., \hat p_M$.

I could imagine some simple model such as:

$$\hat p_i \sim N(p_i, \epsilon^2 + \eta^2 )$$

Where $\eta^2$ is for the measurement error.

Questions:

1. Are there other reasonable models for this type of situation?
2. What are good ways to estimate $\eta^2$ (for the purpose of building confidence interval)? For example, would using $\hat \eta^2 = \frac{1}{n-1}\sum (p_i - \hat p_i)^2$ make sense? Or, for example, it makes sense to first take some transformation of the $p_i$ and $\hat p_i$ values (using logit, probit or some other transformation from the 0 to 1, to an -inf to inf scale)?

Answer 1

I believe there is a simple enough Bayesian solution here. I haven't tried to implement any of this though!

Without measurement error, this is a simple Beta-Bernoulli model:

$$ \begin{align} p_i &\sim \text{Beta}(\alpha_i, \beta_i)\\ y_{i,j} &\sim \text{Bernoulli}(p_i) \end{align} $$

where $p_i$ is the proportion of people who like movie $i$, and $y_{i,j} = 1$ if person $j$ reported liking movie $i$ and $0$ otherwise. $\alpha_i$ and $\beta_i$ are prior parameters, and can be used to encode information about the proportion of people we would expect to like each movie (see here for details on the Beta distribution).

If there is an unknown probability that each person just responds at random for every movie, we can add an additional set of parameters, $r_j$ ($r$ for random) encoding the probability that person $j$ is a random responder, and use Beta priors for each person. I assume that random responders have a 50% chance of responding that they liked the movie.

$$ \begin{align} p_i &\sim \text{Beta}(\alpha_i, \beta_i) \\ r_j &\sim \text{Beta}(a_r, b_r) \\ y_{i,j} &\sim \text{Bernoulli}( r_j \times 0.5 + (1-r_j)\times(p_i)); \end{align} $$

On its own, this model probably isn't identifiable, but fortunately you have a lot of prior information about the first 100 movies. This means you can set strong priors on $\alpha_i$ and $\beta_i$ for $i$ in $1, 2, \dots, 100$, and use non-informative priors (e.g. $\alpha_i = \beta_i = 1$) for the rest of the movies. This information should constrain things enough to have some idea who the random responders are, and to down-weight their responses for the rest of the movies.

This kind of model can be fairly easily fit using standard Bayesian tools like Stan, PyMC3, or JAGS.