1
$\begingroup$

Informal question

Suppose there is a fruit (an apple or an orange) in the next room and ten people observe the fruit. Eight of them report that the fruit is an apple. What is the probability that the fruit is an apple?

Formal question

Let $X\in\left\{1,\ldots,l\right\}$ be the true label of an object and let $N_x$ be the number of independent observations reporting that the label of the object is $x$. What is $$ P\left(X=x|N_1,\ldots,N_l\right)? $$

I have a feeling this is a standard problem in categorical inference but unfortunately, I do not know what to Google for. In the following, I present some of the thoughts I've had about the problem.

Naïve approach

Naïvely, one might assume that $$ P\left(X=x|N_1,\ldots,N_l\right) = \frac{N_x}{N}, $$ where $N=\sum_x N_x$ is the total number of observations. However, the observations are imperfect such that the probability that the true label is $x$ is non-zero even if $N_x=0$ (all observations might have been incorrect).

Slightly more sophisticated approach

Suppose that each observation is an independent Bernoulli trial which succeeds (assigns the correct label) with (known) probability $q$ and fails (assigns the wrong label) with probability $1-q$ . Applying Bayes' theorem I find $$ P\left(X=x|N_1,\ldots,N_l,q\right) \propto P\left(N_1,\ldots,N_l|X=x,q\right), $$ where I have assumed that the prior over true labels is uniform. The likelihood function (and thus the posterior for $x$ given $q$) is binomial such that $$ P\left(X=x|N_1,\ldots,N_l,q\right)\propto q^{N_x}(1-q)^{N-N_x}. $$

Graphical model

In general, the reliability of the observations $q$ is not known a priori. Consider the graphical model in the figure. White circles denote variables; grey boxes denote distributions.

graphical model

The conditional distributions take the form $$ \begin{align} P\left(x|N_1,\ldots,N_l,q\right) &\propto q^{N_x}(1-q)^{N-N_x}\\ P\left(q|N_1,\ldots,N_l,x\right) &\propto q^{N_x}(1-q)^{N-N_x} \Theta\left(q-1/2\right), \end{align} $$ i.e. the distribution for $x$ is a categorical distribution and $q$ is beta-distributed with $\alpha=1+N_x$ and $\beta=1+N-N_x$ but restricted to lie in the interval $[.5, 1]$. The Heaviside function encodes the belief that the observations are at worst random guesses (but not malicious).

In principle, I should be able to sample the parameters using a Gibbs sampler because I have the conditional distributions. Unfortunately, I cannot recover the parameters for simulated data. Any help would be very much appreciated.

$\endgroup$
  • $\begingroup$ Can you add details: What data you simulate, how do you perform the Gibbs sampling and what the results look like? $\endgroup$ – Juho Kokkala Apr 28 '14 at 15:58
  • $\begingroup$ In fact, the Gibbs sampling approach yields correct results. I made a mistake in the analysis of the results. $\endgroup$ – Till Hoffmann Apr 28 '14 at 16:35
1
$\begingroup$

Some observations: if you have more than two classes ($l>2$), your model is not well-defined. If $l=2$, your model can be interpreted as a reparametrization of the standard beta prior + binomial observation model.

Issue with $l>2$: distribution of wrong label not defined

You have not defined which wrong label is issued in the case a wrong label is issued. For example, if the wrong label is picked uniformly from the set of wrong labels, the likelihood should be \begin{equation} P(N_1,\ldots,N_l \mid X=x) = q^{N_x}\,\left(\frac{1}{l-1}(1-q)\right)^{N-N_x} \end{equation} To see that, this matters, consider, e.g. the case with 3 classes, observations $N_1=2,N_2=1,N_3=0$ conditional on $q=0.5$. With the wrong-label-picked-uniformly model, the class probabilities are $(4/7,2/7,1/7)$ while with the likelihood written in your question, they would be $(1/3,1/3,1/3)$.

$l=2$: Reinterpretation as Beta-binomial

Let us define an auxiliary variable $\tilde{q}$ as follows: \begin{equation} \tilde{q} = \left\{ \begin{array}{ll} q, & \quad X=1 \\ 1-q, & \quad X=2 \end{array} \right. \end{equation} Now, given uniform class probabilities $(P(X=1)=0.5)$, $N_1 \mid \tilde{q} \sim \textrm{Bin}(N,\tilde{q})$ and $\tilde{q} \sim U(0,1)$. Therefore, $\tilde{q} \mid N_1,N_2 \sim \textrm{Beta}(1+N_1,1+N_2)$. Furthermore, the event $X=1$ is exactly the same event as $\tilde{q}\geq 0.5$. Thus, $P(X=1 \mid N_1,N_2)$ can be obtained analytically.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.