4
$\begingroup$

Original question

$x_1 , \cdots , x_n$ are independent random variables, identically distributed as a uniform distribution over $(0,\theta)$.

$$ f(x \vert \theta) = \frac{1}{\theta}, \; 0<x<\theta, \;\; \theta >0 $$

What's the Maximum Likelihood Estimator for $\theta$.

Comment on strict inequalities

(from olooney, edited slightly)

The MLE does not exist if we use strict inequality. But $x_i \sim U(0, \theta)$ has various definitions but makes the intent clear: to model the data as a uniform distribution. For a continuous p.d.f. any finite number of points has measure 0 and can be added or removed and the answer is "almost surely" (jargon for "with probability 1") the same.

Why not use a definition of the U which makes the MLE problem tractable and which is almost surely the same?

Note that the question has strict inequalities, not weak

This is highlighted in this answer

Other similar posts to this

Though I searched others have highlighted other posts of a similar nature :

MLE for Uniform $(0,\theta)$

How do you differentiate the likelihood function for the uniform distribution in finding the M.L.E.?

$\endgroup$

closed as off-topic by Xi'an, mkt, Michael Chernick, user158565, StatsStudent Feb 16 at 14:56

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Self-study questions (including textbook exercises, old exam papers, and homework) that seek to understand the concepts are welcome, but those that demand a solution need to indicate clearly at what step help or advice are needed. For help writing a good self-study question, please visit the meta pages." – Xi'an, mkt, Michael Chernick, user158565, StatsStudent
If this question can be reworded to fit the rules in the help center, please edit the question.

3
$\begingroup$

The p.d.f for one $x_i$ is given as

$$ f(x| \theta) = \begin{cases} \frac{1}{\theta} & & \text{if } 0 \leq x \leq \theta \\ 0 & & \text{otherwise} \end{cases} $$ Let's call $\vec{x} = (x_1, ..., x_n)$.

The $n$ observations are i.i.d. so the likelihood of observing the $n$-vector $\vec{x} = (x_1, ... x_n)$ is the product of the component-wise probabilities. Ignoring the issue of support for the moment, note that this product can be simply written as a power:

$$ f(\vec{x}| \theta) = \prod_i^n \frac{1}{\theta} = \frac{1}{\theta^n} = \theta^{-n} $$

Next, we turn our attention to the support of this function. If any single component is outside its interval of support $(0, 1/\theta)$, then its contribution to this equation is a 0 factor, so the product of the whole will be zero. Therefore $f(\vec{x})$ only has support when all components are inside $(0, 1/\theta)$.

$$ f(\vec{x}| \theta) = \begin{cases} \theta^{-n} & & \text{if } \forall i, \ 0 \leq x_i \leq \theta \\ 0 & & \text{otherwise} \end{cases} $$

By definition, this is also our likelihood:

$$ \mathcal{L}(\theta; \vec{x}) = f(\vec{x}| \theta) = \begin{cases} \theta^{-n} & & \text{if } \forall i, \ 0 \leq x_i \leq \theta \\ 0 & & \text{otherwise} \end{cases} $$

The MLE problem is to maximize $\mathcal{L}$ with respect to $\theta$. But because $\theta > 0$ (given in the title of the problem) then $\theta^{-n} > 0$ therefore 0 will never be the maximum. Thus, this is a constrained optimization problem:

$$ \hat{\theta} = \text{argmin}_\theta \,\, \theta^{-n} \text{ s.t. } \forall i \,\, 0 \leq x_i \leq \theta $$

This is easy to solve as a special case so we don't need to talk about the simplex method but can present a more elementary argument. Let $t = \text{max} \,\, \{x_1,...,x_n\}$. Suppose we have a candidate solution $\theta_1 = t - \epsilon$. Then let $\theta_2 = t - \epsilon/2$. Clearly both $\theta_1$ and $\theta_2$ are on the interior of the feasible region. Furthermore we have $\theta_2 > \theta_1 \implies \theta_2^{-n} < \theta_2^{-n}$. Therefore $\theta_1$ is not at the minimum. We conclude that the minimum cannot be at any interior point and in particular must not be strictly less than $t$. Yet $t$ itself is in the feasible region, so it must be the minimum. Therefore,

$$\hat{\theta} = \text{max} \,\, \{x_1,..., x_n\}$$

is the maximum likelihood estimator.

Note that if any observed $x_i$ is less than 0, then $\mathcal{L}$ is a constant 0 and the optimization problem has no unique solution.

$\endgroup$
  • 1
    $\begingroup$ This answer is much nicer than a similar one on math stack, but I'm just wondering if you've used $0 \leq x \leq \theta$ instead of $0 < x< \theta$ on purpose? The original post has strict inequalities. $\endgroup$ – baxx Feb 9 at 16:35
  • 2
    $\begingroup$ That difference is unimportant in practice, both intervals have probability 1 under the model. $\endgroup$ – kjetil b halvorsen Feb 9 at 17:30
  • $\begingroup$ @kjetilbhalvorsen here it seems to be the difference between whether or not the MLE exists though. Are you suggesting that I should ignore that? That's not a loaded question, I'm not familiar enough with the material. Maybe considering strict inequalities here doesn't have much merit, thanks $\endgroup$ – baxx Feb 9 at 21:57
  • 2
    $\begingroup$ @baxx, you are of right - the MLE does not exist if we use strict inequality. But you also wrote $x_i \sim U(0, \theta)$ which has various definitions but makes the intent clear: to model the data as a uniform distribution. For a continuous p.d.f. any finite number of points has measure 0 and can be added or removed and the answer is "almost surely" (jargon for "with probability 1") the same. Why not use a definition of the $U$ which makes the MLE problem tractable and which is almost surely the same? $\endgroup$ – olooney Feb 10 at 15:47
  • $\begingroup$ @olooney fair enough, that seems reasonable. I've added that note to the OP as others might consider the answer an answer to a different question in a similar way to how I did. Hopefully it's alright for me to paraphrase your comment there. Thanks. $\endgroup$ – baxx Feb 10 at 16:25
1
$\begingroup$

Let us first consider the case when the question is read as stated. This is the approach I would recommend if asked this question when, say, taking a class in introductory theoretical statistics. Toy examples like this are commonly assigned as homework problems or on exams in such classes.

You can show (do it) that the likelihood function is $$ L(\theta) = \begin{cases} \theta^{-n} & \theta > x_{(n)} \\ 0 & \theta \leq x_{(n)}\end{cases}, $$ where $x_{(n)} = \max(x_1, \dots, x_n)$. The maximum likelihood estimate (MLE), if it exists, is the $\theta \in (0, \infty)$ that maximizes $L(\theta)$. In this case, since the inequality $\theta > x_{(n)}$ is strict, the MLE does not exist. Indeed, for any $\theta' > x_{(n)}$, the likelihood can always be increased by moving slightly to the left. However, you cannot pick as your MLE $\hat{\theta} = x_{(n)}$ since $L(x_{(n)}) = 0$. Clearly, no points to the left of $x_{(n)}$ can be the MLE either since the likelihood vanishes there.

Now let us also deal with the case where the question is interpreted differently. If you are free to pick which density for the $U(0, \theta)$ distribution to consider, i.e. you do not have to use the one given in the question, then it makes sense to pick the the density $f(x\mid \theta) = \theta^{-1}$, $0 \leq x \leq \theta$, instead. Recall, (Lebesgue) densities are only unique up to changes on sets of Lebesgue measure zero, so this is a density for the same distribution. With this density, the MLE exists and is equal to $\hat{\theta} = x_{(n)}$ which is easy to prove by noting that the likelihood is strictly decreasing on $[x_{(n)}, \infty)$.

$\endgroup$
  • $\begingroup$ yes, $L(\theta) = 1/\theta^n$. We want to maximise this, so that we would choose the lowest possible value of $\theta$. The lowest possible value of $\theta$ that will include all possible values of $x$ is $x_{max}$, so we should say that the MLE of $\theta$ is $x_{max}$. I'm not sure if I'm getting mixed up with the inequalities though. $\endgroup$ – baxx Feb 9 at 16:07
  • 1
    $\begingroup$ You are, that's exactly what my answer says. You have $L(x_{(1)}) = 0$, so it cannot be the MLE, right? You need to be more careful when you state what the likelihood is. $\endgroup$ – Anon Ymous Feb 9 at 16:10
  • 2
    $\begingroup$ @StubbornAtom That would be incorrect, because $\sup(x_1, \dots, x_n) = \max(x_1, \dots, x_n)$. The MLE does not exist as the question is stated, contrary to what the accepted answer claims. This is easy to prove. In fact, I have given this question in several classes to students (which is about the only setting I can think of right now where toy examples like this are practically relevant). $\endgroup$ – Anon Ymous Feb 10 at 17:38
  • 1
    $\begingroup$ @baxx I notice you have now added comments to the answer suggesting that you are equally interested in the case with weak inequality, is that right? If so, I can delete my answer. $\endgroup$ – Anon Ymous Feb 10 at 17:44
  • 1
    $\begingroup$ @baxx I added an explanation of why I think the difference is important. $\endgroup$ – Anon Ymous Feb 10 at 18:21
1
$\begingroup$

As already discussed in the answer by Anon Ymous, the discussion on whether or not using the strict inequality is not pertinent from a measure theoretic point of view: the definition of a $\mathcal U$$(0,1)$ distribution is one of a probability measure against the Lebesgue measure on $\mathbb R$ that gives any open interval within $[0,1]$ a probability proportional to its length:$$\mathbb{P}_\theta[U\in(a,b)]\propto b-a$$ This distribution being absolutely continuous wrt the Lebesgue measure, it enjoys a density$\frac{\text{d}\mathbb{P}_\theta}{\text{d}\lambda}$ that is constant almost everywhere on $[0,\theta]$. Hence $$f_\theta(x)=\frac{1}{\theta}\mathbb I_{(0,\theta)}(x)\quad\text{and}\quad f_\theta(x)=\frac{1}{\theta}\mathbb I_{[0,\theta]}(x)$$ are two equivalent versions of the density of the same distribution. In such instances, the MLE will exist for some versions and not for others, which requires the selection of a smooth version of the density and hence of the likelihood function that allows for a solution. This is obviously a drawback of the MLE approach. (A similar difficulty arises with the Bayesian MAP which further depends on the definition of the dominating measure over the parameter space.)

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.