Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I've been trying to teach myself some of the fundamentals of statistics by trying to work through old qualifying exams. Here's a problem:

Suppose $X_1, \ldots, X_n$ are a random sample from a normal distribution with mean $\theta$ and variance $\sigma^2$, where $\sigma^2$ is fixed and $\theta>0$ is a parameter. Find the maximum likelihood estimator of $\sqrt{\theta}$.

My work so far:

I have the likelihood function $$L(\theta|\mathbf{x})=(2\pi \sigma^2)^{-n/2} \text{exp}\Big(-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i-\theta) \Big).$$

Then, differentiating $L(\theta|\mathbf{x})$ with respect to theta, and equating the result to zero yields $$\sum_{i=1}^{n}(x_i-\theta)=0 \implies \theta=\frac{1}{n}\sum_{i=1}^{n}x_i \hspace{3mm} (=\bar{x}).$$ The second derivative is negative here, hence the MLE of $\theta$ is $\hat{\theta}=\bar{X}$. So, by the invariance property of MLEs, the MLE of $\sqrt{\theta}$ is $\sqrt{\hat{\theta}}=\sqrt{\bar{X}}.$

Question:

My question is about the $\theta>0$ assumption (I'm pretty sure $\sqrt{\bar{X}}$ would be fine if there were no restrictions on $\theta$). So, for $\theta>0$, would the MLE for $\theta$ be something like $\max\{\bar{X},0\}$?

I would greatly appreciate any feedback, corrections, etc.

share|improve this question
2  
If $\theta < 0$, then $\sqrt{\bar{X}}$ is complex-valued with (arbitrarily) high probability as $n$ increases, so in that sense, there is a problem. However, your intuition regarding the MLE on the restricted parameter space is correct. How would you prove it? (Hint: Use the definition of "maximum likelihood" directly.) –  cardinal Jan 10 '13 at 23:14
    
The definition of MLE is just the parameter that maximizes likelihood, right? This seems pretty intuitive to me - we're just looking at the joint probability of our sample and trying to figure out what parameter is most probable. Showing this directly appears to be easy enough. Thanks for the comment! –  pedrosuavo Jan 11 '13 at 2:49

1 Answer 1

So, I'll modify your problem slightly to avoid dealing with boundary issues. Instead of your constraint $\theta > 0$, I'll replace it with $\theta \geq 0$.

You want to maximize the likelihood subject to $\theta \geq 0$.

After taking the logarithm of your likelihood and ignoring constant terms, we get the problem:

$$ \min_{\theta} f(\theta) \text{ s.t. } \theta \geq 0$$

where

$$f(\theta) := \sum_{i=1}^n (X_i - \theta)^2.$$

You are correct that if we didn't have the constraint, we could simply differentiate the objective function and get $\theta^{unconstrained} := \bar{X}$.

However, due to the constraint, we can't just differentiate. So, let us consider the two cases separately:

  1. If $\theta^{unconstrained}$ is positive, then it is also the solution for your constrained MLE problem (the additional constraint can only increase the value of the minimization problem above).

  2. If $\theta^{unconstrainted} < 0$, then it doesn't satisfy your constraint and is not feasible. However, you can check for yourself (a bit of algebra) that $f(\theta) \geq f(0)$ for all $\theta \geq 0$ when $\theta^{constrainted} < 0$. Therefore, $\theta^0=0$ minimizes $f(\theta)$ over $\theta \geq 0$.

So for this problem, the MLE for $\theta$ is $\theta^{ML} = \max{(\bar{X}, 0)}$. And using the equivariance property, the MLE of $\sqrt{\theta}$ is $\sqrt{\theta^{ML}}$.

share|improve this answer
    
This might be a silly question, but why can't we simply just differentiate, then consider the parameter restriction(s)? –  pedrosuavo Jan 11 '13 at 2:53
    
So basically ignore the parameter constraints, maximize (which in this case can be done by differentiation), then somehow project onto the constraint set? I don't think that this will in general give the best value. –  Dapz Jan 11 '13 at 6:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.