5
$\begingroup$

A box contains $5$ white and $2$ black balls. A coin with unknown $P(Head)=p$ is tossed once. If it lands HEADS then a white ball is added, else a black ball is added to the box. Then a ball is selected at random from the box. Given that the ball drawn is WHITE, find the Maximum Likelihood Estimator of $p$.

I find this problem quite confusing, really. It seems to be pretty straightforward and hence I am shocked by the substandard quality, else I am making some serious error. My attempt is as follows:

$P(White)=P(White|Head)P(Head)+P(White|Tail)P(Tail)=\dfrac{6}{8}.p+\dfrac{5}{8}(1-p)=\dfrac{p}{8}+\dfrac{5}{8}$

This is actually my likelihood of $p$ given the sample (my sample is WHITE ball). So this is maximized for $\hat{p}=1$. So $1$ (????) is the MLE for $p$. It is a constant estimator.

This is kind of weird. Any suggestion/correction/explanation is welcome.

$\endgroup$
  • 4
    $\begingroup$ What do you find to be substandard about the problem? Because it is confusing compared to most of the other problems that you have attempted? Because it is pretty straightforward whereas you want to practice on harder problems? The MLE is always some number, that is, a constant. Why do you think it is weird that the number happens to be $1$ in this case? $\endgroup$ – Dilip Sarwate Feb 17 '15 at 17:00
  • $\begingroup$ Well, it is pretty straightforward and came in an exam where it SHOULD NOT have been asked. Besides there is no good mathematics involved. Maybe I am more of a mathematical statistician. Anyway, that the answer is correct is all the confirmation I needed. Of course the MLE can be 1 and my sample was pretty small, so the answer is reasonable. Still, for personal reasons, I would not term this question as a good question. $\endgroup$ – Landon Carter Feb 17 '15 at 17:06
  • 6
    $\begingroup$ One point of such a mathematically straightforward problem is to help you understand the statistical concepts. You should be wondering about this MLE. Does it make sense? For a given loss function, what is its risk? What would other estimation procedures (such as a Bayes procedure) yield? To what extent does the standard MLE theory break down in this case? Far from being "substandard," such a question is perpetually fascinating to those who are interested in the statistical issues and recognize they are distinct from any mathematical developments. $\endgroup$ – whuber Feb 17 '15 at 22:01
  • 3
    $\begingroup$ Is there a real question here? Are you asking for confirmation of your answer? It looks like a perfectly reasonable question to me, and the answer is revealing, especially if you haven't seen such problems (otherwise it's obvious without doing the calculations). You might like to consider the case of drawing more balls (say, two) - what happens when all are white or all are black is (again) obvious enough, but what about when there's some of each? In the case of your estimator, note that your result ($\hat p=1$) applies for your particular sample; it's an estimate rather than an estimator... $\endgroup$ – Glen_b -Reinstate Monica Feb 17 '15 at 22:38
  • 2
    $\begingroup$ The end of my earlier comment seems to have been lost. The point I was coming to at the end there was that consistency applies as $n\to \infty$, so I'm not sure it makes sense to discuss consistency unless we're discussing an estimator which we've defined for all $n$; currently it's defined for one particular experiment (there's one observation, that the ball was white). You'd want to consider a case where many observations were generated in some way and have an estimator for that case, whose consistency could then be discussed. $\endgroup$ – Glen_b -Reinstate Monica Feb 18 '15 at 10:13
2
$\begingroup$

Let $B$ denote an event of probability $p$. Then, the law of total probability says that $$\begin{align} P(A) &= P(A\mid B)P(B) + P(A\mid B^c)P(B^c)\\ &= P(A\mid B)\cdot p + P(A\mid B^c)\cdot (1-p) \end{align}$$ showing that $P(A)$ is a linear function of $p$, having value $P(A\mid B^c)$ when $p=0$ and value $P(A\mid B)$ when $p=1$. For $p\in (0,1)$, the value of $P(A)$ is somewhere between these extreme values. Thus, for $p \in [0,1]$, the maximum value of $P(A)$ is either $P(A\mid B)$ or $P(A\mid B^c)$ (except, of course, when $P(A\mid B) = P(A\mid B^c)$ -- which means that $A$ and $B$ are independent events -- and also means that $P(A)$ has the same value for all $p \in [0,1]$: knowledge that $A$ occurred is of no help in making inferences about the occurrence of $B$ or the value of $p$).

In this instance, $B$ is the event of tossing a Head on the coin and $A$ the event of drawing a White ball. Since $P(A\mid B) = \frac 68$ and $P(A\mid B^c) = \frac 58$ we have that $P(A)$ has maximum value $\frac 68$ when $p = 1$.

$\endgroup$
2
$\begingroup$

Your maximum likelihood estimate is correct. A more careful formalization may help your understanding of the concepts involved. In the following, take notice of how conditional independence is used.

Let $X=0,1$ be the result of the experiment, with $X=0$ meaning "Black ball is drawn", and $X=1$ meaning "White ball is drawn". Introduce a parameter $\Theta\in[0,1]$, and a random variable $Y$ such that $Y\mid\Theta=\theta\sim\mathrm{Bernoulli}(\theta)$, with $Y=1$ standing for "Heads".

The experiment can be formalized specifying the distribution of $X$ given $Y$ as $$ P(X=0\mid Y=0) = \frac{3}{8} \, , \qquad\qquad P(X=1\mid Y=0) = \frac{5}{8} \, , $$ $$ P(X=0\mid Y=1) = \frac{1}{4} \, ,\qquad\qquad P(X=0\mid Y=1) = \frac{3}{4} \, , $$ and postulating that $X$ and $\Theta$ are conditionally independent, given that $Y=y$.

Using the law of total probability and the product rule, the likelihood is $$ L_x(\theta) = P(X=x\mid \Theta=\theta) = \sum_{y=0,1} P(X=x, Y=y\mid \Theta=\theta) $$ $$ = \sum_{y=0,1} P(X=x\mid Y=y ,\Theta=\theta)\,P(Y=y\mid \Theta=\theta) $$ $$ = \sum_{y=0,1} P(X=x\mid Y=y)\,P(Y=y\mid \Theta=\theta) \, , $$ in which the last equality follows from the postulated conditional independence. Hence, the likelihood for your data is $$ L_1(\theta) = \frac{5}{8} \cdot (1-\theta) + \frac{3}{4} \cdot \theta = \frac{\theta}{8} + \frac{5}{8} \, , $$ and $\hat{\theta}_{\text{ML}}=1$.

The problem looks artificial because you are trying to estimate the parameter of the Bernoulli with just one observation.

Following Huber's suggestion, you may do a Bayesian analysis using the prior $\Theta\sim\mathrm{Beta}(a,b)$. The posterior distribution is a mixture of two betas. Can you find the Bayes estimate with quadratic loss?

$\endgroup$
  • 1
    $\begingroup$ No I cannot :( My exams are coming up and I will have to wait for at least another week before I can deviate from the current syllabus. As of now, the risk function is not covered. $\endgroup$ – Landon Carter Feb 18 '15 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.