4
$\begingroup$

This question was on a HW in my Statistical Theory class and I find the professor's answer and explanation to be unsatisfactory. Please give me some guidance as to why

  • $\bar{x}$ is the MLE if this is the case, or

  • Let me know if I am correct to think that both $\bar{x}$ and $1-\bar{x}$ maximize the likelihood function and therefore the MLE is not unique, or

  • If the problem is even well defined as it is posed. I feel like this could be the case also.

I understand the definition of the MLE in more regular circumstances, as detailed in the Principles section of http://en.wikipedia.org/wiki/Maximum_likelihood, but the weird form of the PDF brings up issues with supremums that I am not used to dealing with.

Question: Let $X_1,...,X_n$ be an i.i.d. sequence of 0-1 valued RV's with the probabilities

$$ P(X_1=1)=\begin{cases} \theta, & \theta\in\mathbb{Q}\\1-\theta, & \theta\notin\mathbb{Q} \end{cases} $$

where $\theta\in(0,1)$. Does the MLE of $\theta$ exist?

Outline of Professor's solution: This is the main idea of my professors solution. The likelihood function is

$$ L(\theta|x_1,...,x_n)=\{\theta^{\sum{x_j}}(1-\theta)^{n-\sum{x_j}}\chi_{\theta\in\mathbb{Q}} +\theta^{n-\sum{x_j}}(1-\theta)^{\sum{x_j}}\chi_{\theta\notin\mathbb{Q}}\} $$

where $\chi_A$ is the indicator function of the set $A$. We have

$$ \begin{eqnarray} \underset{\theta\in[0,1]}{\sup} L(\theta|x_1,...,x_n)&=&\underset{\theta\in[0,1]}{\sup} {\{\theta^{\sum{x_j}}(1-\theta)^{n-\sum{x_j}}\chi_{\theta\in\mathbb{Q}} +\theta^{n-\sum{x_j}}(1-\theta)^{\sum{x_j}}\chi_{\theta\notin\mathbb{Q}}\}} \\&=& \max\{\underset{\theta\in\mathbb{Q}}{\sup} \{\theta^{\sum{x_j}}(1-\theta)^{n-\sum{x_j}}\},\underset{\theta\notin\mathbb{Q}}{\sup} \{\theta^{n-\sum{x_j}}(1-\theta)^{\sum{x_j}}\}\} \\&=& \max\{\bar{x}^{\sum{x_j}}(1-\bar{x})^{n-\sum{x_j}},(1-\bar{x})^{n-\sum{x_j}}\bar{x}^{\sum{x_j}}\} \\&=& \bar{x}^{\sum{x_j}}(1-\bar{x})^{n-\sum{x_j}} \end{eqnarray} $$

Professor: At this point, the professor argues that the supremum in the second term,$\underset{\theta\notin\mathbb{Q}}{\sup} \{\theta^{n-\sum{x_j}}(1-\theta)^{\sum{x_j}}\}\}$ is not attained since $\bar{x}$ is a rational number. Since the data consists of rational numbers, the supremum of the first term , $\underset{\theta\in\mathbb{Q}}{\sup} \{\theta^{\sum{x_j}}(1-\theta)^{n-\sum{x_j}}\}$ is attained at $\hat{\theta}_1=\bar{x}$ and that this is the MLE of $\theta$.

Me: It seems like we should consider the supremum over the closure of the sets $\mathbb{Q}$ and $\mathbb{R}\backslash\mathbb{Q}$, which would be $[0,1]$, in which case $\sup L(\theta|x_1,...,x_n)$ is achieved at both $\hat{\theta}_1=\bar{x}$ and $\hat{\theta}_2=1-\bar{x}$. Otherwise, we are essentially assuming that $\theta$ is rational and ignoring irrational $\theta$. Is this the case? If so, is this an undesirable property of the Likelihood Principle in weird cases like this? Is there any plausible situation where issues like this occur? Should I stop worrying about weird problems like this?

As an aside, considering $[0,1]\backslash\mathbb{Q}$ has Lebesgue measure 1 and $[0,1]\cap\mathbb{Q}$ has Lebesgue measure 0, it seems like $\bar{x}$ is a bad estimator, since it is an estimate of $\theta$ if it is in a very small set. Also, if $\theta\in\mathbb{R}\backslash\mathbb{Q}$, $\hat{\theta}_2=1-\bar{x}$ is consistent, so I can't think of a good reason why $\bar{x}$ is better.

Edit As @cardinal pointed out, the $x_i$ are obviously rational, so this is not an issue. This addressed my first (silly) misunderstanding, which involved assuming the estimator $\bar{x}$ could be irrational or rational.

$\endgroup$
2
  • 2
    $\begingroup$ The $X_i \in \{0,1\}$, so, irrespective of everything else, $\bar X$ is certainly rational! $\endgroup$
    – cardinal
    Nov 26, 2012 at 18:00
  • $\begingroup$ There is a good book that I think has the solution to this particular problem. It's called The Cauchy-Schwartz Master Class. I cannot recall their solution off of the top of my head. $\endgroup$
    – R S
    Dec 10, 2012 at 3:08

1 Answer 1

1
$\begingroup$

I am not a professor of Statistics, and so most probably my approach will be naive.

Since we want to consider cases where the parameter characterizing the distribution belongs to disjoint sets of numbers, it follows that they will be different numbers. So the notion "the MLE of $\theta$" appears to be wrong or at least confusing/misleading, since it will not be the same $\theta$... Perhaps a more transparent definition of the probability mass function would be

$$P(X_1=1)=p=\begin{cases} \theta_1, & p\in\mathbb{Q}\\1-\theta_2, & p\notin\mathbb{Q} \end{cases}$$

with $0< \theta_1\neq \theta_2<1$.
In the one scenario where $p$ cannot be rational, while at the same time the MLE can only take rational values, we are effectively forcing the range of the estimator to have no common elements with the parameter space where $p$ lives. But of course, this is a special case of "empty intersection", since the values that the MLE takes are "very near-by" the values $p$ can take, and this is why the MLE is a consistent estimator of $p$, a property that is probabilistic and does not require the deterministic limit of the MLE to converge to the true $p$. Note that the MLE cannot be strongly consistent, since the event $\{\hat p_{MLE} = p\}$ is impossible by construction. So it only converges in probability, not almost surely.

Now, if $p$ could be any real number in the interval $(0,1)$, then the likelihood $L(p|\mathbf x)$, as a function of $p$ would be maximized at $p=\bar x$ (second-order conditions for a maximum are also satisfied). This means that

$$ L(p=\bar x) > L(p\neq \bar x)$$

In other words, when $p$ cannot be rational, then the MLE $\hat p = \bar x$ makes the likelihood acquire a value that is higher than any value it can take for any non-rational value of $p$, and for any conceivable sample from a $\{0,1\}$ random variable. By continuity, I believe that this means that $\hat p= \bar x$ gives us the supremum of the function "Likelihood given $p$ is not rational" (although it is not its maximum, since the function "Likelihood given $p$ is not rational" is not defined for a rational number).

So, if the OP accurately conveyed the argument of the professor, ("in the $p$- cannot-be-rational case, the supremum is not attained since $\bar x$ is a rational number"), it appears that I have just laid down an argument to the exact opposite effect.

When the likelihood is the function "Likelihood given $p$ is rational", then the sample mean is the argmax of this function, and so it gives us its supremum=maximum also.

So in both cases, we are good in obtaining the maximum likelihood estimator of $p$ as the sample mean of the data.

Then, if it is revealed to us that we are in the scenario "$p$ is rational", we have already obtained the MLE of $\theta_1$. If it is revealed to us that we are in the scenario "$p$ is not rational" then, by using the invariance property of the MLE we have $\hat \theta_2 = 1-\hat p_{MLE} = 1-\bar x$.

$\endgroup$
3
  • $\begingroup$ Thanks for looking at this! First, I think it is important that in the definition of the PMF that $\theta_1 = \theta_2$. That way, the supremum for $p\in \mathbb{Q}$ is $\bar{x}$, the supremum for $p\in \mathbb{R} \ \ \mathbb{Q}$ is $1 - \bar{x}$, and the suprema are equal. Then the only difference is whether the supremum over the set is attained, which occurs for rational $p$, but not irrational $p$. Since we have no knowledge of what set $p$ is in, we look at both suprema, see that it is only obtained for rational $p$, ignore irrational $p$, and set $\hat{p}=\bar{x}$. $\endgroup$
    – caburke
    Jul 10, 2014 at 16:59
  • $\begingroup$ Basically, the problem becomes an uninteresting exercise in analysis because we have to restrict the domain of the estimator to $\mathbb{Q}$, even though "most" parameter values are in $\mathbb{R} \\\mathbb{Q}$. It is about as interesting as saying the recursion $a_{n+1} = \frac{1}{a_n} + \frac{a_n}{2}$ has no limit over $\mathbb{Q}$, even though it converges to $\sqrt(2)$ over $\mathbb{R}$. The main lesson is that probabilists should not teach statistics. $\endgroup$
    – caburke
    Jul 10, 2014 at 17:12
  • $\begingroup$ I agree with your comments, although I still cannot understand how we are allowed to state that the two parameters are equal while at the same time they belong to disjoint sets. $\endgroup$ Jul 14, 2014 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.