3
$\begingroup$

Let $\mathbf{X} = (X_1, \ldots, X_n)$ be an i.i.d sample from the parametric family of distributions $\mathcal{P} = \{P_\theta: \theta \in \Theta \subset \mathbb{R}^k\}$ ($X_i \sim P_\theta$ are random variables, not just real numbers). In my statistics textbook there is the following formula for the Maximum Likelihood Estimator $\hat\theta(\mathbf{X})$ (which is also a random variable, not an estimate):

$$\hat\theta(\mathbf{X}) = \underset{\theta \in \Theta}{\operatorname{argmax}} L(\mathbf{X}; \theta),$$ where $L(\mathbf{X}; \theta)$ is a random variable which is obtained from an ordinary (numerical) likelihood function $L(\mathbf{x},θ)$ by substituting random vector $\mathbf{X}$ on place of numerical vector $\mathbf{x}$.

I am concerned about the fact that random vector $\mathbf{X} = (X_1, \ldots, X_n)$ implicitly depends on $\theta$ because $X_i \sim P_\theta$. This makes it difficult to interpret the maximization process.

Should I treat random variables $(X_1, \ldots, X_n)$ as constants in the maximization process?

If not, what will be the correct interpretation of the above formula?

$\endgroup$
  • 1
    $\begingroup$ The concept of MLE is: assuming the observed $\pmb{x}$ is "most-likely" to occur, we wish to find the value of $\theta$ which maximizes the likelihood $L(\pmb{x};\theta)$. Thus the MLE of $\theta$ will be a function of $\pmb{x}$. Now for further analysis, if we wish to find distribution or confidence interval of the MLE of $\theta$, we replace $\pmb{x}$ by $\pmb{X}$. $\endgroup$ – Shanks May 19 '18 at 21:18
1
$\begingroup$

To make the distinction between $\theta$ which is the running argument in the likelihood function $L({\mathbf X};\theta)$ and $\theta_0$ which is the parameter at the source of the generation of $\mathbf X$ [assuming the model is true and there is such a $\theta_0$], one can write $\mathbf X$ as the deterministic transform$${\mathbf X}={\mathbf \Psi}(\epsilon,\theta_0)\sim P_{\theta_0}$$where $\epsilon$ is a "white noise" in the sense it is a random vector with a fixed distribution, of dimension possibly larger than the dimension of $\mathbf X$ (to keep the representation as general as possible). The likelihood function then writes as$$L({\mathbf \Psi}(\epsilon,\theta_0);\theta)$$which can be optimised in $\theta$ [and not in $\theta_0$ since, while ${\mathbf X}={\mathbf \Psi}(\epsilon,\theta_0)$ is observed, $\epsilon$ is not.

$\endgroup$
  • 1
    $\begingroup$ Thanks! You mean that I should write $X_i \sim P_{\theta_0}$ (where $\theta_0$ is true parameter) instead of $X_i \sim P_\theta$ in my post above to use the formula for $\hat\theta(\mathbf{X})$? It seems that in this case $\mathbf{X}$ will not depend on varying parameter $\theta$ and the formula will be correct. $\endgroup$ – Rodvi May 20 '18 at 13:20
  • 1
    $\begingroup$ (+1) Exactly so! This "tale of two $\theta$'s" is also at the core of the EM algorithm. $\endgroup$ – Xi'an May 20 '18 at 13:57
0
$\begingroup$

Although Xi'an's answer is correct I want to note that in most cases we use model where random vector $\mathbf{X}=(X_1, \ldots, X_n)$ doesn't depend on any parameter at all, only its distribution depends on $\theta$. In such cases the maximization process is easily interpreted. You can find details here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.